Indonesia: Decentralized Basic Education Project · the decentralized basic education sector were...

Performance Evaluation Report

Indonesia: Decentralized Basic Education Project

EvaluationIndependent

Reference Number: PPE:INO 2014-15 Loan and Grant Numbers: 1863-INO and 0047-INO

Independent Evaluation: PE-774

Performance Evaluation Report Performance Evaluation Report Performance Evaluation Report Performance Evaluation Report NovemberNovemberNovemberNovember 2014201420142014

IndonesiaIndonesiaIndonesiaIndonesia: : : : Decentralized Basic Education Project Decentralized Basic Education Project Decentralized Basic Education Project Decentralized Basic Education Project

This document is being disclosed to the public in accordance with ADB's Public Communications Policy 2011.

NOTES

(i) The fiscal year of the government ends on 31 December. (ii) In this report, “$” refers to US dollars. (iii) For an explanation of rating descriptions used in ADB evaluation reports, see

Independent Evaluation Department. 2006. Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. Manila: ADB (as well as its amendment effective from March 2013).

Director General V. Thomas, Independent Evaluation Department (IED) Director W. Kolkma, Independent Evaluation Division 1, IED

Team leader H. Son, Principal Evaluation Specialist, IED Team member S. Labayen, Associate Evaluation Analyst, IED

The guidelines formally adopted by the Independent Evaluation Department on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of the management of the Independent Evaluation Department, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgment as to the legal or other status of any territory or area.

Abbreviations ADB – Asian Development Bank BOS – Bantuan Operational Sekolah

(School Operational Assistance Program) DBEP – Decentralized Basic Education Project MONE – Ministry of National Education MORA – Ministry of Religious Affairs NTB – Nusa Tenggara Barat (West Nusa Tenggara) NTT – Nusa Tenggara Timur (East Nusa Tenggara) PCR – project completion report TA – technical assistance

Currency Equivalents Currency Unit – rupiah (Rp)

At Appraisal At Project

Completion At Independent Evaluation (15 October 2001) (15 June 2012) (31 August 2014)

Rp1.00 = $0.0001 $0.0002 $0.0001 $1.00 = Rp9,935 Rp9,420 Rp11,705

Contents Acknowledgments vii

Basic Data ix

Executive Summary xi

Chapter 1: Introduction 1 A. Evaluation Purpose and Process 1 B. Expected Results 2 C. Evaluation Method 2

Chapter 2: Design and Implementation 4 A. Formulation 4 B. Rationale 5 C. Cost, Financing, and Executing Arrangements 6 D. Procurement, Construction, and Scheduling 8 E. Outputs 8 F. Consultants 9 G. Covenants 10 H. Policy Framework 10

Chapter 3: Performance Assessment 12 A. Overall Assessment 12 B. Relevance 12 C. Effectiveness 19 D. Efficiency 27 E. Sustainability 28

Chapter 4: Other Assessment 30 A. Impact 30 B. ADB and Executing Agency Performance 30

Chapter 5: Issues, Lessons, and Follow-up Actions 32 A. Issues 32 B. Lessons 33 C. Follow-up Actions 34

APPENDIXES 1 Summary Design and Monitoring Framework

36

2 List of Interviewees 41 3 Questionnaire for Schools 44 4 Number and Type of Project Schools 50 5 Rating Matrix for Core Evaluation Criteria 51 6 Education Statistics 53 7 Poverty Statistics 55

Acknowledgments A team of staff and consultants from the Independent Evaluation Department (IED) contributed to this study by conducting data gathering and analysis, desk reviews, interviews, research, and surveys. Hyun H. Son (team leader) prepared this report and Stella Labayen (associate evaluation analyst) formatted it. The report benefited from the overall guidance of Vinod Thomas and Walter Kolkma. The team acknowledges the valuable inputs of Rizza Leonzon and Ekki Syamsulhakim, the consultants who worked on this evaluation. The team also thanks Wolfgang Kubitzki, Ferny Suhandi, and Imelda Marquez for their assistance in organizing meetings with central and district government officials, development partners, and project schools. The team is grateful to Asian Development Bank (ADB) staff, school representatives, and the Government of Indonesia for their hospitality, as well as assistance and participation in the interviews. Their insights into the decentralized basic education sector were most helpful in writing the report. The report was peer reviewed by Tu Chi Nguyen of the World Bank and Farzana Ahmed of IED. Valuable comments were received on an earlier draft from ADB’s Southeast Asia Department and Indonesia Resident Mission. The report was also shared with the Government of Indonesia, which welcomed the findings and recommendations of the study.

Basic Data Decentralized Basic Education Project (Loan 1863-INO; Grant 0047-INO) Key Project Data ($ million)

As per ADB Loan/Grant Documents Actual

Total Loan Cost 125.0 138.4 Foreign Exchange Cost 21.1 26.3 Local Currency Cost 103.9 112.1

ADB Financed 100.0 112.1 Borrower Financed 25.0 26.3

Total Grant Cost 36.3 29.5 Foreign Exchange Cost 36.3 29.5

Borrower Financed 8.3 1.5 Government of the Netherlands 28.0 28.0 Key Dates Expected Actual Appraisal 6–26 July 2001 Loan Negotiations 3–10 October 2001 Board Approval

- Loan - Grant

29 November 2001

14 March 2006 Loan Agreement 20 February 2002 Loan Effectiveness 20 May 2002 20 May 2002 Closing Date

- Loan - Grant

31 December 2008 31 December 2008

30 June 2009

30 September 2011 Closing Extensions 2 (Grant) Loan Effectiveness to Closing (months) 81 87 Project Completion June 2012 Borrower Government of Indonesia Executing Agency Ministry of National Education Mission Data Type of Mission No. of Missions No. of Person-Days Inception 1 138 Handover 1 14 Review 12 309 Special project administration 2 82 Midterm 1 75 Final 1 10 Project completion review 1 20 Independent evaluation 1 9

Executive Summary Decentralization is a policy option that has been adopted by many countries to improve access to basic services, including education. Since 2001, the Government of Indonesia has embarked on a program to decentralize delivery of basic education. By making the education system more attuned to local needs, decentralization—the process of delegating authority or functions from central government to local government units—will be able to play a crucial role in improving the access to and quality of basic education in Indonesia. To help Indonesia roll out its decentralized basic education program, the Asian Development Bank (ADB) supported a project to improve the participation in and completion of 9 years of basic education, particularly among the poor. The project included primary and junior secondary levels in the provinces of Bali, East Nusa Tenggara (NTT), and West Nusa Tenggara (NTB). It supported three outcomes: (i) improved participation, transition, completion, and performance in basic education among poor children in Bali, NTB, and NTT; (ii) implementation of school-based management in project schools; and (iii) effective district education management in the three provinces. To achieve these objectives, the project had three components: school development; district basic education development; and monitoring, evaluation, and reporting. The project was financed by a combination of loan and grant. Of the $138.4 million loan component, $112.1 million was financed by ADB and $26.3 million by the Government of Indonesia. The grant component totaled $29.5 million, of which $28.0 million was funded by the Government of the Netherlands and $1.5 million by the Government of Indonesia. This report presents performance evaluation findings of the Decentralized Basic Education Project based on four core evaluation criteria (relevance, effectiveness, efficiency, and sustainability) and two additional criteria (institutional development and impact). Overall, the project performance is rated successful. The project is rated highly relevant. The timing of its implementation was responsive to addressing insufficient capacity during Indonesia’s transition to a decentralized basic education system. The project is also aligned with the education policies and strategies of both the government and ADB. Changes in the project coverage involving withdrawal of five Jakarta districts and redirection of this funding to two districts in impoverished NTT improved its relevance. Modifications in project design during the early phase of implementation, as well as weak coordination between the Ministry of National Education (MONE) and the Ministry of Religious Affairs (MORA), the project’s executing and co-executing agencies, respectively, posed challenges to implementation but did not compromise relevance either in terms of targeting schools and districts or in achieving the objective of improving access of the poor to basic education. The project is effective. Analysis of primary time-series data obtained during the evaluation indicates that statistically significant improvements in education outcomes

xii Indonesia: Decentralized Basic Education Project

including enrollment, completion, transition, and performance were observed in selected project districts after the project was implemented. The project also helped to improve school capacity by promoting an evidence-based approach to formulating school development plans; encouraging community involvement in education matters; and improving district capacity for basic education, particularly in teacher development and the evaluation of school development plans. The project is efficient. Despite the limited size of grants, the project supported notable achievements in improving school and district capacity for managing basic education and enhancing school infrastructure. The progress of its loan component was initially slow, but this picked up throughout project execution. The grant component was twice extended, as the implementation period of only 2.5 years proved too short for severely poor and often inaccessible districts. The project is likely sustainable. The formulation of school development plans, which was pioneered by the project, has become standard practice among schools. School committees continue to participate in creating school strategies and budget plans. School and district officials who received training under the project continue to impart knowledge to their counterparts who were not part of the project. The central and district governments provide the primary source of funding to help project schools deliver continued support for poor students through scholarships, transport allowances, remedial and catch-up programs, and tutoring for students missing school. The project is rated as having a significant impact on enhancing local capacity for basic education management, improving education outcomes, and shaping government policies related to decentralized basic education services. ADB’s performance is rated less than satisfactory, given that high turnover of its project officers constrained effective and continued supervision and management of the project. MONE’s performance is rated satisfactory, since it eventually gained sufficient capacity after initial difficulties. The evaluation draws the following lessons: i. Decentralization of basic education can be reinforced through strong

collaboration between MONE and MORA. Structural misalignment hampers the coordination of these ministries in managing basic education because MORA remains centralized while MONE has been decentralized. Although both ministries may have misaligned structures and varying levels of resources and expertise, they are nevertheless jointly tasked with providing basic education. A more collaborative effort is needed for its successful delivery and management.

ii. Good technological infrastructure at the school and district levels is imperative

for data management. Although there is an online database for school indicators, called Dapodik, schools have yet to fully maximize this medium because many lack internet access. Capacity building of school staff is also necessary for the online system to work effectively.

iii. High staff turnover impedes capacity building and disrupts efforts to promote

decentralized basic education management. Frequent staff changes among ADB’s project officers and district governments hampered the transfer and sharing of knowledge, as it did communication and rapport-building among stakeholders.

Executive Summary xiii

Given these lessons, the project performance evaluation report makes these recommendations: i. Coordination between MORA and MONE, which is crucial for decentralized

basic education, requires improvement. Best practices in education management need to be shared with officials of both ministries. Moreover, the role of provincial governments in decentralized basic education should be strengthened, particularly in monitoring and evaluation. This is because MORA’s provincial offices are primarily tasked with providing madrasah education, while responsibility for general education lies with district governments and district MONE offices.

ii. Endeavors to enhance the delivery of basic education in Indonesia should be tied in with continued investments in enhancing transport and technological infrastructure and school facilities. Improving the access of the poor to transport is a priority everywhere, but particularly in remote areas. In these areas, improving access to junior secondary schools, which are normally located in subdistrict capital cities, is especially needed. Improvements in technological infrastructure in schools and districts are necessary to facilitate continuous data updates on the Dapodik system. Continued rehabilitation of classrooms in remote areas in Bali, NTB, and NTT is also necessary.

iii. More than a decade since it was first implemented, decentralized basic education in Indonesia remains beset with challenges. These include its impact on teacher policy, given the considerable district authority in staffing (including over transfer of teachers) and the perceived politicization of education appointments made by district governments. Should ADB choose to continue providing support in this area, it should consider how such challenges could be best addressed.

CHAPTER 1

Introduction A. Evaluation Purpose and Process 1. This project performance evaluation report assesses the Decentralized Basic Education Project, which was designed to support decentralized planning, management, and delivery of basic education services in Indonesia. The project aimed to enhance participation in and completion of 9 years of basic education, particularly among the poor in Bali, West Nusa Tenggara (NTB), and East Nusa Tenggara (NTT). The project was evaluated based on four core criteria (relevance, effectiveness, efficiency, and sustainability) and two additional criteria (institutional development and impact), as specified under Asian Development Bank (ADB) guidelines. 1 The evaluation also assesses the project’s impact on student performance in beneficiary districts, analyzes the roles of its stakeholders, and reviews other factors that may have affected project performance. This evaluation’s findings will feed into the forthcoming validation of Indonesia’s County Partnership Strategy 2012–2014 final review. 2. The project completion report (PCR) was prepared in June 2012. Overall, the project was rated successful based on the four core evaluation criteria. It was rated highly relevant to the poverty reduction priorities of ADB and the Government of Indonesia. Changes in the project’s scope—which included adding districts in impoverished NTT—further improved its relevance. The project was rated effective, based on (i) improvements in net enrollment, completion, and transition rates and examination scores (although no targets had been specified); (ii) implementation of school-based management in 3,000 schools and the establishment of effective community-school partnerships; (iii) increased district education funding over 2001 levels; (iv) development of data-based district education development plans; and (v) implementation of plans for quality and equity improvement. It was rated efficient because progress under the loan component picked up during implementation despite initial delays. It was rated partly sustainable. The project helped to mainstream school and district planning and school-based management through national policies. As noted in the PCR, however, some stakeholders took the view that activities financed under the project would not be sustainable without project support. 3. The project was assessed to have a likely significant impact on project districts and provinces given improved school management that involved both school officials and communities. The PCR noted, however, that it was difficult to assess the project’s impact on district capacity.

1 Independent Evaluation Department. 2006. Guidelines for Preparing Performance Evaluation Reports for

Public Sector Operations. Manila: ADB (as well as its amendment effective from March 2013).

2 Indonesia: Decentralized Basic Education Project

B. Expected Results

4. The project sought to support the decentralized planning, management, and delivery of basic education services to improve participation in and completion of 9 years of basic education. It was implemented in 26 districts across 3 provinces (Bali, NTB, and NTT).2 The project targeted 5,238 schools in poor villages, of which 4,329 were primary and 909 were junior secondary schools. It supported three outcomes: (i) improved participation, transition, completion, and performance in basic education among poor children in the three provinces; (ii) implementation of school-based management in project schools; and (iii) effective district education management in the three provinces. 5. To achieve its objectives, the project had three components: school development; district basic education development; and monitoring, evaluation, and reporting. It supported the following major outputs: (i) implementation of school-based management, (ii) improved school quality with School Development Fund assistance, (iii) improved capacity of districts to plan and manage decentralized basic education as reflected in their 5-year district education development plans, (iv) improved district education with District Education Development Fund assistance, and (v) effective program and financial monitoring compliance. The project’s design and monitoring framework is presented in Appendix 1. C. Evaluation Method

6. The evaluation used qualitative and quantitative techniques in examining the project’s performance. The qualitative approach was used for in-depth interviews with officers from the Ministry of National Education and district education offices, as well as school principals, teachers, and staff involved in the project. Appendix 2 provides a complete list of government and school officials interviewed during the evaluation mission. Appendix 3 presents the questionnaire for school officials. 7. The quantitative approach was particularly useful for examining education outcomes and poverty status. Various secondary data were used in analyzing education outcomes, including those for enrollment, transition, and completion rates; national exam scores; and poverty status in selected provinces and districts. The evaluation also used the findings of a survey conducted during the mission that provided primary time-series data on education indicators and other information related to the project. Survey questionnaires were distributed to selected project schools. The quantitative approach used in this evaluation study makes before-and-after comparisons. For some quantitative outcomes, such as enrollment, completion, and transition rates as well as exam scores, the analysis involved mean-difference testing to check robustness of the comparison analysis. 8. The survey’s sample of project schools was selected using a mixture of probabilistic and non-probabilistic approaches to ensure that each school in each province had an equal opportunity to be selected as part of the sample. The sample for this survey covered 56 schools, which are representative in terms of districts and types of schools.

2 Jakarta was initially included as one of the three target provinces. However, it was withdrawn from the

project because its income was sufficient to achieve objectives similar to those of the project. The funds previously allocated to Jakarta were then redirected to NTT.

Introduction 3

9. Although not the case of Bali, schools in NTB and NTT are scattered on various islands within the two provinces. Hence, the study purposively selected the islands within these provinces. Lombok Island in NTB and Timor Island in NTT were chosen. The numbers of districts within the islands was then calculated proportionately, resulting in two districts each for Bali and NTT and three districts for NTB. A simple random sampling mechanism was used to select the districts on each of the islands. On that basis, the selected districts were Bangli and Buleleng in Bali; Lombok Barat, Lombok Timur, and Mataram in NTB; and Timor Tengah and Timor Tengah Utara in NTT. Taking into account differences in the local education policies, as well as socioeconomic characteristics in each of the provinces and districts, this study calculated proportionately the number of project schools to be surveyed for each district. This calculation was based on the number and types of schools covered by the project as presented in the PCR (Appendix 4). A stratified random sampling mechanism based on the level of school was then used to decide which project schools were selected as part of the sample.

CHAPTER 2

Design and Implementation A. Formulation

10. The Government of Indonesia has implemented numerous measures to improve the access to and quality of basic education, including decentralizing the management and delivery of basic education following the passage of Law No. 22/1999 on Regional Governance and Law No. 25/1999 on Fiscal Decentralization. To this end, district governments have been tasked with providing financing for basic education.

11. In 1997, the government requested ADB support for improving the delivery of basic education in the provinces of Bali and West Nusa Tenggara (NTB) and the Special Capital Region Jakarta (footnote 2). The government selected these provinces because of their gaps in basic education and the regional demarcation between ADB and the World Bank.3 Jakarta withdrew from the project, however, and was replaced by East Nusa Tenggara (NTT). 12. ADB in 1998 approved small-scale project preparatory technical assistance (TA), but the Asian financial crisis caused project preparation to be suspended.4 This was resumed in 2000. The government requested that the project support the decentralization of basic education management to build on achievements of the ADB-funded Social Protection Sector Development Program. ADB subsequently approved small-scale TA to redesign the project.5 A loan appraisal mission was fielded during July 2001. 13. The project had no attached TA, but two related TA projects and a small project funded by the Japan Fund for Poverty Reduction complemented project implementation. Bridging TA providing support for decentralized education management commenced implementation in March 2002 and supported the initial steps in decentralizing the management of basic education to improve service delivery. This included capacity building for a special decentralization unit in the Ministry of National Education (MONE).6 Additional TA on integrating poverty considerations into decentralized education management began implementation in April 2003, shortly after project consultants were fielded. The TA supported poverty mapping for 16 districts covered by the project and 50 participatory poverty studies.7 The project,

3 ADB. 2001. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to

the Republic of Indonesia for the Decentralized Basic Education Project. Manila. 4 ADB. 1998. Technical Assistance to the Republic of Indonesia for Basic Education in Bali and Nusa Tengara

Barat Project. Manila (TA 3007-INO, $150,000, approved 16 April 1998). 5 ADB. 2000. Technical Assistance to the Republic of Indonesia for the Preparation of Decentralized Basic

Education Project. Manila. (TA 3456-INO, $150,000, approved 14 June 2000). 6 ADB. 2006. Technical Assistance Completion Report: Support for Decentralized Education Management in

Indonesia. Manila (TA 3701-INO). 7 ADB. 2007. Technical Assistance Completion Report: Integration of Poverty Considerations in Decentralized

Education Management in Indonesia. Manila (TA 3957-INO).

Design and Implementation 5

backed by Japan Fund for Poverty Reduction, was approved in May 2002 and consultants were fielded 12 months later.8 It pilot tested school-based management and community-based education as a pro-poor service delivery scheme in particularly underserved schools. 14. The project was funded by a $100.0 million loan from ADB and grant cofinancing of $28.0 million from the Government of the Netherlands. The loan was approved on 29 November 2001 and closed on 31 December 2008. The grant was approved on 14 March 2006 and extended for 1 year, to 31 December 2009, to allow for project completion in NTT, and later to 31 December 2010 to cover emergency reconstruction work in Sumatra. Some $10.0 million was reallocated from the loan to rebuild schools after an earthquake and tsunami struck Aceh and North Sumatra in December 2004. Similarly, grant savings of $1.3 million were used in reconstructing 9 schools in West Sumatra after a 2009 earthquake. B. Rationale 15. The project provided crucial support during Indonesia’s transition to a decentralized system of basic education. Its commencement coincided with the initial implementation of Law No. 22/1999 on Regional Governance and Law No. 25/1999 on Fiscal Decentralization in 2001, during which a lack of technical guidance for provincial and district governments in executing the laws, especially for the education sector, posed a challenge. The project therefore supported pioneering efforts in implementing a decentralized education system.9 16. Per the government’s request, the project supported the process of decentralizing basic education management, in accordance with the laws on regional governance and fiscal decentralization (footnote 3). Prior to implementation of these laws, the education system was highly centralized. Basic education was managed centrally through complex and compartmentalized structures in three ministries: the Ministry of Home Affairs, MONE, and the Ministry of Religious Affairs (MORA). MONE was responsible for overall planning, coordination, and regulation for education at all levels and it directly administered and financed public junior secondary education. The Ministry of Home Affairs administered and delivered public primary education, overseeing teacher management, establishment and maintenance of primary schools, and provision of subsidies to private primary schools. MORA administered and financed public primary and junior secondary madrasah schooling and supervised and provided financial assistance to private madrasahs. This system had been replicated at provincial and district levels, where all three ministries were represented in parallel by local offices. Similar to this complex management structure, budgeting and financing for basic education involved five ministries: the Ministry of Finance, Ministry of Home Affairs, MONE, MORA, and the National Development Planning Agency, with a mix of financing arrangements and funds sources for each school type and level. 17. Under Indonesia’s decentralized basic education system, district governments are now responsible for planning, financing, managing, and delivering basic education. Their responsibilities also cover school staff appointments and transfers and school location planning. The central government sets policies, the national curriculum, and a national examination and assessment system. It also sets standards for (i) achievement at each age; (ii) learning materials; (iii) acceptance, transfer, and certification of 8 ADB. 2007. Implementation Completion Memorandum for INO: Supporting Community-Based Basic

Education (JFPR 9016). Manila. 9 Study team’s interviews with stakeholders during evaluation mission.

The project provided crucial support during Indonesia’s transition to a decentralized system of basic education


students; and (iv) learning times for basic education and fixing the educational calendar. The central government establishes procedures and standards for staff appointments and dismissals; determines pensions, salaries, benefits, and rights and obligations; and establishes the legal status of civil servants and regional civil servants, including teachers and principals. 18. Provincial governments (i) define policy on student selection and acceptance, including equity for minority students and students from poor families and remote areas; (ii) provide instructional materials for basic education; (iii) assist in higher education management, curriculum production, accreditation, staff appointments, and opening and closing of colleges; and (iv) manage special institutions, including training centers, teacher training institutions, and special education. Provincial governments may also decide on the transfer between districts of civil servants, including school principals and teachers, and their transfer between cities and provinces (footnote 3). 19. By supporting the decentralized system, the project also responded to challenges in improving the equity in and quality of delivering basic education services.10 The proportion of primary school-age children enrolled at this level—the net enrollment rate—was 93.0% at the start of project implementation in 2001, while the ratio of female-to-male primary enrollment was 98.0%. Nearly 660,000 primary school-aged children were out of school in that year, more than 60.0% of them girls.11 20. ADB’s development agenda for Indonesia also prioritizes basic education. Until 1990, ADB support for Indonesia’s education sector was channeled mainly to the vocational, technical, and higher education subsectors. Since then, however, ADB has increased support for basic education to improve poor children’s access to basic education and enhance learning outcomes at this level. Among ADB’s basic education investments in Indonesia is the Social Protection Sector Development Program, which supported a scholarship and school block grant program. The Social Protection Sector Development Program was introduced in the aftermath of the Asian financial crisis. It helped maintain stability in primary and junior secondary school enrollment and supported policy dialogue for improving social safety nets mechanisms. C. Cost, Financing, and Executing Arrangements 21. At appraisal, the total cost of the loan component was $125.0 million. At loan closing, the actual project cost was $138.4 million, of which $112.1 million was financed by ADB and $26.3 million by the government. The appreciation of special drawing rights versus the dollar caused the 11.0% increase in the loan amount. There were no cost overruns or underruns since project expenditures were linked to school and district development plans. 22. In terms of cost categories, the PCR noted that expenditure on research, surveys, and studies (42.0% of the appraisal estimate) was low because the project had not carried out baseline and midterm evaluation surveys and other studies. 12 The evaluation team reviewed two impact evaluation documents produced under the project: (i) Decentralized Basic Education Project: Consulting Services for Baseline and Benefit Impact/Evaluation Studies, Initial Findings (2006); and (ii) Benefit, Monitoring,

10 Indonesia’s education system consists of preschool, primary education (grades 1 to 6), junior secondary

education (grades 7–9), senior secondary education (grades 10–12), and tertiary education. Basic education covers 9 years of compulsory primary and junior secondary education.

11 ADB. 2013. Key Indicators for Asia and the Pacific. Manila. 12 ADB. 2012. Completion Report: Indonesia: Decentralized Basic Education Project. Manila (p. 7).


and Evaluation Studies of the Decentralized Basic Education Project (Baseline and Midterm Evaluation Studies): Final Report (2008).13 However, these impact evaluation studies did not clearly identify the type of impact measurement being calculated. Moreover, only baseline and midline evaluations were conducted. The baseline study did not incorporate the appropriate control group as the respondents at the school level only included principals, teachers, students, and school committees from 55 project schools. While the midterm report stated that it included non-project schools, it did not sufficiently discuss how the non-project schools were selected. Moreover, the baseline report used the district minimum service standard as the comparison benchmark, which may result in misleading findings since the minimum service standard may also include project schools. Given the different sampling methods and questionnaires used by the baseline and midterm evaluation studies, the study team assumed there to be no connection between the two. The study team also had no access to the data used in the studies. As the PCR had noted, due to the absence of data, information for measuring project benefits also relied on the memories of the stakeholders.14 23. The estimated cost of the grant component was $36.3 million, of which $28.0 million was to be funded by the Government of the Netherlands and $8.3 million by the Government of Indonesia. The actual cost was $29.5 million, of which the Netherlands funded $28.0 million and Indonesia $1.5 million. 24. At loan closing, the project had disbursed $112.1 million, about 112% of the original loan amount and 97.0% of the current loan value, which included an initial advance of $2.6 million (25 June 2002) to finance the cost of various small contracts. The project disbursed $28.0 million in grant funds, including the ADB administration cost of $0.5 million. The loan account was financially closed on 30 June 2009 and the grant on 30 September 2011. 25. MONE was the project’s executing agency. MORA was the co–executing agency and assigned staff to project management offices at the different levels of government. At national level, a project steering committee provided policy guidance. District coordination committees, education boards, and school committees assisted the steering committee. Meanwhile, the central technical committee assumed technical and programmatic functions and the central facilitation unit managed overall financial and administrative coordination. The facilitation unit was supported by provincial and district project management units. Although complex, this implementation structure was appropriate given the project’s large size and constant changes in administrative context (footnote 3).

13 The project’s baseline and midterm evaluations can be accessed in ADB’s Archives Section. See Insan

Hitawasana Sejahtera. 2008. Decentralized Basic Education Project: Consulting Services for Baseline and Benefit Impact/Evaluation Studies, Initial Findings. Jakarta; and Indonesia Ministry of National Education and Indonesia University of Education. 2006. Benefit, Monitoring and Evaluation Studies of the Decentralized Basic Education Project (Baseline and Midterm Evaluation Studies) Final Report. Jakarta.

14 Proper impact evaluations should include appropriate design and identification of control and treatment groups. The design of an impact evaluation should be specified at the beginning of a program and include the type of measurement impact (e.g., average treatment effect, average treatment on the treated, spillover), sampling method of both treatment and control groups, and the evaluation method. The selection of treatment and control groups should also be adequately discussed. Impact evaluations should be done also at the baseline and endline of a project.


D. Procurement, Construction, and Scheduling 26. The loan component was approved in 2001, took effect in 2002, and closed on schedule in 2008. Although there were initial delays, the pace of implementation picked up. Some variations in project implementation procedures were adopted. Phased implementation was originally identified at the planning stage, whereby the batches were based on schools within districts rather than on the districts themselves. Phased implementation was abandoned, however, because districts wanted to commence project activities immediately. Given this change, all original districts commenced project activities in mid-2003 and all districts continued to receive support over the next 5 years. Within these districts, three different batches of schools also received support for 3 years. Challenges were also encountered in use of the School Development Fund, which was released a year before consultants were mobilized and before schools had formulated their school development plans. 27. The grant was approved in 2006 and extended twice before it was closed in 2010 due to (i) a 1-year extension to make sure that the programs benefited NTT, and (ii) a second 1-year extension to allow grant savings to be used for reconstructing schools in Padang Pariaman. 28. Civil works involved rehabilitation of existing facilities as well as construction of earthquake-resistant schools on the remote island of Simeulue and rehabilitation of earthquake-ravaged schools in Padang Pariaman. 29. Schools and districts carried out most of the procurement for items eligible under the District Education Development Fund and the School Development Fund. District project management units procured eligible items under the district education fund through direct purchases using limited local tenders. On the other hand, schools procured eligible items under the School Development Fund through direct purchase and school committees managed procurements for the rehabilitation of schools. Cash expenditures used direct purchase and were supplemented by such in-kind contributions from the community as materials and labor. Training and workshops were provided by facilitators and trainers according to ADB and government guidelines. 30. Several procurement problems were uncovered by the external auditor, most of which were related to overpayment for workshops and training. As noted in the PCR, the findings of the external audit indicate that community monitoring of school activities and finances is a viable method of ensuring compliance. E. Outputs 31. The project supported three major components: (i) school development; (ii) district basic education development; and (iii) monitoring, evaluation, and reporting. These components were supported by the following outputs. 32. Implementation of school-based management. The project improved the schools’ capacity in developing and implementing school development plans. Project schools were required to develop these plans to access project funds for scholarships for the poor, rehabilitation of classrooms and other facilities, and other activities identified in their plans. About 97% of the 4,244 project schools drew up these plans,


according to the baseline study.15 However, project schools were not required to report their achievements against their plans. The project also helped to strengthen community involvement in schools through the formation of school committees. 33. Improved school quality with School Development Fund support. This was the project’s funding mechanism to support activities identified in school development plans. To ensure transparency and accountability of the School Development Fund, its accounts were audited annually by external auditors. Public accountability systems were also established to monitor the project’s operation. As noted in the PCR, 95% of schools reported on the School Development Fund, program and budget plans, and program and budget implementation. Moreover, 85% of schools were reported to have established complaint mechanisms, though only 10% of school principals were seen as willing to receive complaints. 34. Improved capacity of districts as reflected in a 5-year district education development plan. The project established district education boards in compliance with Ministerial Regulation No. 044/2002. These were tasked with monitoring district education programs, as well as school development plans. The boards were provided training on education management, including budget analysis. The legal framework for the boards was revised in 2010 by a new regulation on the management of education (PP 17/2010). As a result, funding of the boards by districts became discretionary and the number of board members was reduced. Although board members work in a voluntary capacity, they nevertheless require funding for their operations. The project also required all districts to publish their education statistics annually and to complete their district education development plans. During the project, all districts published statistics annually and completed their district education development plans. Some districts continue to update their plans and publish their statistics, but weaknesses in information systems make it difficult to keep the data current and for them to be able to publish the statistics on time. 35. Improved district education with District Education Development Fund support. To improve district education, the project supported the preparation, implementation, and monitoring of strategic 5-year district education development plans and 5-year annual action plans by all districts. Under the plans, districts developed systems for planning, managing, financing, staffing, and monitoring teacher development; school supervision; recruitment, training, and performance review of school principals; facilities expansion; and district financing of education. The fund supported the implementation of priority district education development programs and capacity building activities. 36. Effective program and financial monitoring compliance. The project helped promote transparency and accountability as part of school-based management. Schools inform communities of their school development plans, programs, and budget plans, as well as their School Development Fund and District Education Development Fund accounts. F. Consultants 37. Under the loan component, three packages of consulting services were envisaged during appraisal: (i) school-based management; (ii) district capacity building

15 Insan Hitawasana Sejahtera. 2008. Decentralized Basic Education Project: Consulting Services for Baseline

and Benefit Impact/Evaluation Studies, Initial Findings. Jakarta.


and project implementation support; and (iii) independent monitoring, evaluation, and reporting. Some 82 person-months of international consultant inputs (compared with 78 person-months at appraisal) and 8,403 person-months of national consulting inputs (compared with 7,782 person-months at appraisal) were used. The consultants for the loan component were mobilized a year after project activities commenced. 38. Under the grant component, two additional packages were planned: (i) school and district building and project implementation support, and (ii) independent monitoring and evaluation. The grant packages mobilized only national consultants, using 1,288 person-months of national consulting inputs (against the planned 960 person-months). All consultants were recruited in accordance with ADB’s Guidelines on the Use of Consultants. Several changes requested by the consulting firm led to considerably greater person-months and cost. A university research institute was contracted in 2006 to conduct the baseline and midterm evaluation study, but it performed poorly. The institute was replaced by a social survey firm in 2007, which conducted the required baseline and impact studies. 39. District project and school personnel considered package (i) and (ii) consultants under the loan component helpful or very helpful, but they found package (iii) consultants too focused on monitoring schools’ utilization of the School Development Fund. The consultants for package (iii) only monitored financial compliance in the schools and districts. They did not conduct program monitoring and evaluation after this activity was removed from their contract. The monitoring and evaluation activities for the loan project did not refer to the design and monitoring framework indicators. Meanwhile, as noted by the PCR, the consultants under the grant component performed well and achieved good results. G. Covenants 40. Twenty-five of 27 covenants were complied with. The loan covenants on the role of the provincial project monitoring units and gender were only partially complied with. These units were envisaged to monitor or evaluate project activities and impact but failed to do so. As a result, only progress and performance were monitored, not the achievement of indicators and targets. The provinces were originally assigned to conduct performance monitoring and evaluation, but this was not carried out. An independent contractor was hired to conduct an impact study on beneficiary schools under the loan project and baseline and impact studies on beneficiary schools in NTT under the grant. Meanwhile, the covenant on gender required two women members per school committee. In NTT, almost 70% reached this target, but results in Bali and NTB were poor. H. Policy Framework

41. The project influenced national government policy and procedures, and most notably the School Operational Assistance Program (BOS), which was launched in 2005. BOS partly adopted the project’s block grant system, and it provides block grants to all schools to enhance the implementation of school-based management and lower the cost of providing basic education. While compulsory fees from parents may not be retained, the central government permits local governments and parents to financially


support schools if BOS funds are insufficient.16 The disbursement of BOS funding is decided by school committees in a similar procedure established by the project. 42. The project also influenced legally binding national education standards introduced from 2006 to 2008 and minimum service standards for districts introduced in 2010. Moreover, school-based management is now official policy and implemented in all schools nationwide. However, the school-managed model for rehabilitation under the project has been discontinued under new legislation requiring a return to district contracting. Moreover, many districts now restrict parental and community financial support for schools, which was one of the project’s achievements.

16 J. Tobias et al. 2014. Towards Better Education Quality: Indonesia’s Promising Path. London: Overseas

Development Institute.

CHAPTER 3

Performance Assessment A. Overall Assessment 43. Overall, the project is rated successful. The overall assessment is based on equally weighted individual assessment criteria: relevance, effectiveness, efficiency, and sustainability (Table 1), which is in accordance with the Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. This rating is generally consistent with that of the PCR, but the rating for sustainability differs somewhat (Appendix 5). Other criteria, such as impact and ADB and borrower performance, are discussed, but not formally assessed because of difficulties in precise attribution and quantification.

Table 1: Assessment of Project Performance

Criterion Weight

(%) Assessment Rating Value

Weighted Rating

Relevance 25 Highly relevant 3 0.75 Effectiveness 25 Effective 2 0.50 Efficiency 25 Efficient 2 0.50 Sustainability 25 Likely sustainable 2 0.50 Impact Significant Institutional Development Satisfactory ADB Performance Partly satisfactory Borrower Performance Satisfactory Overall Rating 100 Successful 2.25

ADB = Asian Development Bank. Note: Highly Successful (>2.7), Successful (2.7 ≥ S > 1.6), Less than Successful (1.6 ≥ LS > 0.8),

Unsuccessful (<0.8). Source: Study team.

B. Relevance 44. The project is rated highly relevant. As stated in the rationale section, the project and the timing of its implementation responded to challenges Indonesia faced during its transition to a decentralized basic education system, including weak institutional capacity; limited human and financial resources; and unclear delineation of roles, functions, and authority of central and local government agencies. The project’s components are therefore deemed appropriate to help address these gaps by empowering schools and district governments in identifying and addressing education needs in their communities. 45. The project also remained relevant despite legislative changes. Fiscal and administrative decentralization took effect on 1 January 2001. The government passed the National Education System Law in 2003 and the National Development Planning

Performance Assessment 13

94.3

96.6

93.6

95.5

85

90

95

100

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Perc

ent

B a l i Nusa Tenggara Barat

Nusa Tenggara Timur Indonesia

System Law in 2004. Between 2002 and 2004, it passed other laws relevant to decentralization, such as those on regional government and central–regional financial balance, the national planning and financial system (budgeting), and the national treasury (payments mechanism). 46. The project’s objective of improving basic education outcomes, including participation, transition, completion, and performance, was also relevant to Indonesia’s education challenges at the time of project implementation. Although Indonesia has nearly achieved universal access to primary education, participation in secondary education remains a concern. The average net enrollment rate of children aged 7–15 years was 94% in primary school and 59% at the secondary level at around the time of project implementation in 1998–2002.17 Figures 1 and 2 indicate that average net enrollment in Nusa Tenggara Timur (NTT) was generally lower than the national average for both primary and secondary levels during 2003–2013, while net enrollment in Bali and Nusa Tenggara Barat (NTB) was slightly higher for much of the period. Low average enrollment in NTT may be attributed to the poor quality of basic infrastructure and school facilities, which restrict children’s access to education services. The slump in Indonesia’s net enrollment rate from 2010 to 2011 coincides with a decrease in public spending on education. Public spending on education reached 3.5% of GDP in 2009 but decreased to 3.0% and 2.8% of GDP in 2010 and 2011, respectively.18 As discussed, the government selected the provinces based on education gaps and the regional demarcation between the Asian Development Bank (ADB) and the World Bank.19

Figure 1: Primary Net Enrollment in Selected Provinces, 2003–2013

Note: Since 2007, the net enrollment rate includes nonformal education (Package A, which is primary school, and Package B, which is junior secondary). Source: Study team calculations based on Indonesian Statistics Office data.

17 National Development Planning Agency. 2002. National Program for Indonesian Children: Education.

Jakarta. 18 Figures are World Bank estimates accessed at data.worldbank.org on 6 October 2014. 19 The World Bank at the time supported basic education projects in the provinces of Central Java, East Java,

Kalimantan, Nusa Tenggara Timur, Sulawesi, Sumatra, and West Java, and was preparing a new basic education project covering East Java and South and West Sumatra. See ADB. 2001. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to the Republic of Indonesia for the Decentralized Basic Education Project. Manila.


80.780.2

59.2

73.7

0

30

60

90

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

B a l i Nusa Tenggara BaratNusa Tenggara Timur Indonesia

47. Improving enrollment at junior secondary level is also an area of concern in Indonesia (Figure 2). Several factors influence low enrollment at the junior secondary level, including household welfare, gender, religious background, child’s performance, supply-side readiness, and employment opportunity. Children from poor households are less likely to enroll in junior secondary school, as are girls and children from Muslim backgrounds. Children with higher national exam scores in primary school are more likely to enroll in junior secondary school. Enrollment at junior secondary level is also found to be positively affected by the number of schools built in those areas where the children live. Finally, children tend to continue their school through junior secondary level if there are few employment opportunities in the community.20

Figure 2: Junior Secondary Net Enrollment in Selected Provinces, 2003–2013

Note: Since 2007, the net enrollment rate includes nonformal education (Package A, which is primary school, and Package B, which is junior secondary). Source: Study team calculations based on Indonesian Statistics Office data.

48. Children dropping out of school also undermined efforts to improve Indonesia’s basic education system. About 600,000 Indonesian children dropped out of primary school in 2001, at the start of project implementation, putting the dropout rate at 2.7%.21 Among junior secondary school students in 2001, the dropout rate was 3.5%, meaning that about 300,000 students left junior secondary school in that year. NTB and NTT had higher dropout rates in 2001 than the national average, at 3.2% and 5.4%, respectively, while Bali had a lower rate of 2.1% (Table 2). In Indonesia, poor children tend to give up schooling for working. Some evidence nevertheless suggests that banning children from working could worsen their school attendance since they will not have sufficient financial resources to continue studying.22

20 Suryadarma et al. 2006. Causes of Low Secondary School Enrollment in Indonesia. SMERU. 21 National Office for Research and Development. 2002. Education Statistics. Ministry of National Education.

Jakarta. 22 A. Suryahadi, A Priyambada, and S. Sumarto. 2005. Poverty, School, and Work: Children during the

Economic Crisis in Indonesia. Development and Change. 36 (2). pp. 351–373.

Perc

ent


Table 2: Dropout rate of Primary and Junior Secondary Students in Selected Provinces, 2001/2002 (%)

Province Primary Junior Secondary Bali 2.1 1.9 Nusa Tenggara Barat 3.2 2.9 Nusa Tenggara Timur 5.4 3.4 Indonesia 2.7 3.5

Source: National Office for Research and Development. 2002. 49. The project was particularly relevant to improving the transition rate of junior secondary school-aged children in NTT. Some of the poorest villages are located near the border of Timor-Leste. According to interviews with stakeholders, the nearest junior secondary school was about 15 kilometers from these villages. In addition, the roads leading to the subdistrict capital, where junior secondary schools are normally located, were in very poor condition, making parents reluctant to send their children to junior secondary schools. Through the project, these districts received assistance to build junior secondary schools, such as those in Biboki Utara and Bikome Utara, allowing children to continue their studies at the junior secondary level. 50. The project was also relevant to the need at the time of implementation for rehabilitating school infrastructure. During 2001/2002, 24% of primary school buildings in Indonesia, including classrooms, showed signs of severe damage and 33% of slight damage (Appendix 6, Table A6.3). About 4% of junior secondary buildings were badly damaged and 10% slightly damaged (footnote 17). In NTT, Bali, and NTB, respectively, 66%, 55%, and 50% of primary classrooms were damaged. It was difficult to ascertain the project’s contribution to improving school facilities at the time of the evaluation mission because other sources of funding were made available for school building rehabilitation after the project’s life. Interviews with Ministry of National Education (MONE), school, and district officials revealed that the government has been rehabilitating schools since 2012, allocating as much as Rp17.5 trillion (equivalent to $1.5 billion) for this endeavor. 51. The project was also relevant to government and ADB education policies and strategies. As discussed earlier, it was highly aligned with the country’s decentralization strategy, which serves as a means for improving the quality of and access to basic education by building a more integrated education system at local level that can effectively respond to local needs and conditions. ADB’s Country Assistance Plan, 2001–2003 for Indonesia acknowledged decentralization of basic education as a major area of ADB support, particularly in addressing the need to develop local management capacity and to ensure better provision of basic education for the poor. The ADB strategy also recognized that although decentralization could improve transparency, accountability, and community participation, there were risks, particularly during the transition period. These included inadequate technical capacity and human and financial resource constraints that could hamper the efficient delivery of basic services, including education. The strategy stated that ADB will help redefine the role and support the reorganization of MONE in line with the decentralization law that was being newly implemented at that time.23 52. Decentralization in Indonesia has attracted support from many donors. These include, among others, ADB, United Nations agencies, and the World Bank. Alongside the government, these three launched a partnership in April 2000 to support governance reforms, including decentralization. ADB assumed a major role in

23 ADB. 2000. Country Assistance Plan: Indonesia, 2001–2003. Manila.


decentralization and various anticorruption initiatives. The project has been the model after which other donor-funded projects on decentralized basic education in Indonesia were patterned. The United States Agency for International Development implemented a similar project which was patterned after the project. ADB’s Country Assistance Plan, 2001–2003 for Indonesia also stated that ADB, UNICEF, and the World Bank shared sector support in education on a geographic basis (footnote 3). For instance, the project excluded Flores Island in NTT because the Australian government was already implementing a primary education project in that area at the time. 53. The project’s scope was relevant to the objective of improving the quality of and access to basic education, particularly among the poor. The changes in scope, which saw five districts in Jakarta withdraw and the corresponding funding reallocated to two districts in NTT, made the project even more relevant. NTT and NTB are among the poorest provinces in Indonesia (Table 3).

Table 3: Poverty Incidence and Annual Changes in Poverty Rates in Indonesia (%)

Province

Poverty Incidence Average Annual Change (2006–

2011) 1996 2006 2011 (%) Rank (%) Rank (%) Rank (%) Rank

Jakarta 2.4 1 4.6 1 3.8 1 3.6 32 Bali 7.8 2 7.1 2 4.2 2 8.1 3 South Kalimantan 8.5 3 8.3 3 5.3 3 7.3 8 Bangka Belitung 10.9 5 5.8 4 9.5 1 Banten 9.8 4 6.3 5 7.1 10 Central Kalimantan 13.5 4 11.0 6 6.6 6 8.1 5 East Kalimantan 9.7 6 11.4 8 6.8 7 8.1 4 Riau Islands 12.2 11 7.4 8 7.8 6 Riau 12.6 8 11.9 10 8.5 9 5.7 19 North Sulawesi 17.9 7 11.5 9 8.5 10 5.3 25 West Kalimantan 24.2 13 15.2 17 8.6 11 8.7 2 Jambi 14.6 5 11.4 7 8.7 12 4.8 28 West Sumatra 9.8 9 12.5 12 9.0 13 5.5 21 North Maluku 12.7 13 9.2 14 5.6 20 South Sulawesi 16.7 11 14.6 15 10.3 15 5.9 17 West Java 11.1 10 14.5 14 10.7 16 5.3 24 North Sumatra 13.2 12 15.0 16 11.3 17 4.9 27 Indonesia 17.6 17.8 12.5 5.9 West Sulawesi 20.7 19 13.9 18 6.6 11 East Java 22.1 16 21.1 21 14.2 19 6.5 13 South Sumatra 15.9 15 21.0 20 14.2 20 6.4 14 Southeast Sulawesi 29.2 20 23.4 25 14.6 21 7.5 7 Central Java 21.6 17 22.2 22 15.8 22 5.8 18 Central Sulawesi 22.3 21 23.6 26 15.8 23 6.6 12 DI Yogyakarta 18.4 14 19.2 18 16.1 24 3.2 33 Lampung 25.6 18 22.8 23 16.9 25 5.1 26 Bengkulu 16.7 19 23.0 24 17.5 26 4.8 29 Gorontalo 29.1 29 18.8 27 7.1 9 Aceh 12.7 23 28.3 28 19.6 28 6.2 15 Nusa Tenggara Barat 32.0 22 27.2 27 19.7 29 5.5 23 Nusa Tenggara Timur 38.9 24 29.3 30 21.2 30 5.5 22 Maluku 44.6 25 33.0 31 23.0 31 6.1 16 West Papua 41.3 32 31.9 32 4.6 31 Papua 42.3 26 41.5 33 32.0 33 4.6 30 Sources: United States Agency for International Development and DAI/Nathan Group. 2013. Provincial Poverty Rates in Indonesia, 2006–2011. Jakarta.

The project’s scope was

relevant to the objective of

improving the quality of and

access to basic education, particularly

among the poor


54. Although Bali may have lower poverty compared to other Indonesian provinces, some of its districts and villages continue to face rampant poverty.24 In the district of Buleleng, for example, some 37,644 people remained poor in 2010. Similarly, the percentage of poor in the district of Jembrana was 8.1% in 2010, compared to Bali’s 4.9% average in the same year. Meanwhile, 21.6% of NTB and 23.5% of NTT were deemed poor in 2010 (Figures 3, 4 and 5).

Figure 3: Percentage of Poor in Bali’s Districts, 2010

Source: Secretariat of the National Team for the Acceleration of Poverty Reduction, Office of the Vice President of the Republic of Indonesia. 2011.

Figure 4: Percentage of Poor in Nusa Tenggara Barat Districts, 2010

Note: North Lombok is a new district, gaining its autonomy in 2008. It was previously part of the district of West Lombok. Source: Secretariat of the National Team for the Acceleration of Poverty Reduction, Office of the Vice President of the Republic of Indonesia. 2011.

24 Secretariat of the Acceleration of Poverty Reduction. 2011. Regional Welfare Indicator: Province of Bali.

Jakarta (Office of the Vice President of the Republic of Indonesia).

8.1 7.9 7.6 7.4 6.9 6.7 6.4

3.22.2

4.9

0123456789

% o

f po

or

District

43.1

23.8 21.8 21.7 21.6 19.9 19.8 19.414.4 12.8

21.6

05

101520253035404550

% o

f po

or

District


Figure 5: Percentage of Poor

in Nusa Tenggara Timur Districts, 2010

Source: Secretariat of the National Team for the Acceleration of Poverty Reduction, Office of the Vice President of the Republic of Indonesia. 2011.

55. Although the project’s design was modified in the early stages of implementation, it remained relevant to the project’s objective of improving poor people’s access to basic education. The four-tier criteria for school selection (for primary and junior secondary schools and madrasah) were simplified to three tiers, in which subdistricts, villages, and schools were ranked. The targeting criteria were eventually simplified further for both the loan and grant to a one-tier procedure based on the poverty ranking of schools within each district. Similarly, demand-led financing was modified, and all schools were given standard amounts of block grant for rehabilitation and quality improvement. 56. These changes in design did not appear to compromise project relevance, particularly in targeting. MONE first selected the poorest districts and then the poorest subdistricts within these impoverished districts. In selecting schools, district education officials conducted site visits to check the actual conditions on the ground and the level of community involvement. Interviews with MONE, school, and district officials indicated that schools in areas where community involvement needs improvement were often considered candidates as project schools. MONE pointed out that the Ministry of Religious Affair’s (MORA’s) weak regulation of madrasah posed challenges during project implementation, noting MORA’s lack of clear guidelines for the establishment of new madrasah. 25 According to interviews with stakeholders, since the project provided grants that a school could use depending on its need, some new schools were

25 During the study team’s interviews, MONE said it follows certain criteria in establishing public schools,

examining first the number of prospective students and the number of students already absorbed by existing schools. For example, there should be at least three primary schools in an area to allow for the construction of a junior secondary school. Moreover, the distance between the new and existing junior secondary school should be 5 kilometers. Unless existing schools do not meet minimum service standards or cannot accommodate the number of students, the request for a new school is rejected. In contrast, according to interviews with MONE officials, Islamic schools were built even if they were just 100 meters away from an existing one.

41.1

34.032.8 32.4 31.7

29.928.7

26.7 25.9

22.9 22.721.6 21.2 20.8 20.4

15.513.4 12.7 12.1

10.6 9.6

23.5

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0


built in order to access the block grant. In Lombok, anecdotal evidence reveals that one primary Islamic school split into two. To address this problem, MONE issued a policy that only schools already existing for 5 years can access the block grant. 57. In selecting poor students, schools usually obtained student socioeconomic data during enrollment, including household income and parents’ employment status. Students who are orphans are often accorded priority, according to interviews with stakeholders. School officials also conducted house visits to verify students’ living conditions. Since the grants provided under the project are relatively small, it was imperative to be highly selective of the project districts and schools, as the interviews reveal. 58. The project’s design was also able to accommodate emergency needs for school infrastructure rehabilitation in the aftermath of the earthquake and tsunami that hit Aceh and North Sumatra in December 2004. As discussed in Chapter 2, loan and grant savings were used to rebuild damaged schools. About 42 schools were reconstructed in the affected areas and a model for earthquake-resistant school buildings was developed.

C. Effectiveness 59. The project is assessed as effective. This evaluation found evidence that education outcomes, including enrollment, completion, transition, and performance in selected beneficiary districts, improved after project implementation. Interviewed stakeholders deemed enhanced school capacity to formulate school development plans and greater community involvement in education concerns to be the project’s most important contributions. 60. Improved education outcomes. Access to basic education improved in the beneficiary provinces (Table 4). Between 2001 and 2011, net enrollment rates at primary level increased in NTB and NTT but slightly decreased in Bali. Primary net enrollment in NTT and Bali remained slightly below the national average in 2011. Net enrollment at the secondary level increased in all three provinces during 2001–2011. NTT’s secondary net enrollment increased from 38.0% in 2001 to 60.9% in 2011 but remained below the national average over the last decade. 61. Completion rates have also improved. NTT saw a notable increase in the secondary completion rate from 92.8% in 2001 to 97.7% in 2011. However, both primary and secondary completion rates in NTT and NTB remained below the national average in 2011, while Bali’s were higher than the national average in that same year (Table 5).


Table 4: Net Enrollment Rates in Selected Provinces, 2001–2011 (%)

Item 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Bali Primary 93.02 92.68 91.73 93.67 93.36 93.33 94.02 94.78 92.18 90.91 91.07 Secondary 69.19 67.97 69.12 70.75 69.75 70.15 73.97 67.63 75.54 77.10 73.27 Nusa Tenggara Barat Primary 91.74 92.65 92.20 91.58 93.32 94.50 95.47 94.50 94.71 95.65 94.15 Secondary 56.95 60.28 61.21 61.08 69.38 69.62 67.00 71.66 77.84 79.49 75.29 Nusa Tenggara Timur Primary 88.52 87.44 88.88 90.68 92.16 91.58 91.57 92.01 89.61 91.34 91.58 Secondary 38.00 40.89 41.16 44.22 43.30 47.23 38.68 49.71 51.20 52.15 60.89 Indonesia Primary 92.88 92.70 92.55 93.04 93.25 93.54 93.66 94.15 91.76 91.55 92.00 Secondary 60.47 61.69 63.49 65.24 65.37 66.52 66.13 67.26 67.97 69.67 69.83 Source: Study team’s calculations based on national socioeconomic survey data.

Table 5: Completion Rates in Selected Provinces, 2001–2011 (%)

Item 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Bali Primary 98.61 98.56 98.77 98.94 98.94 98.83 99.12 99.28 99.01 99.26 99.49 Secondary 97.83 97.57 98.16 97.77 97.32 96.90 99.14 93.12 99.07 99.27 98.90 Nusa Tenggara Barat Primary 98.35 98.26 97.59 97.90 98.77 98.57 98.76 99.16 99.91 99.20 98.89 Secondary 92.62 94.79 94.89 95.48 97.01 97.22 97.29 95.60 96.94 97.79 96.81 Nusa Tenggara Timur Primary 97.70 97.52 97.74 98.19 98.62 98.38 97.82 98.88 98.00 98.46 98.72 Secondary 92.77 97.11 96.40 97.30 97.31 97.17 97.88 95.37 95.98 98.00 97.73 Indonesia Primary 98.54 98.27 98.54 98.68 98.92 98.70 99.04 99.18 98.97 98.84 99.05 Secondary 96.16 97.09 97.18 97.50 96.92 97.11 97.08 95.66 97.01 97.50 96.96 Source: Study team’s calculations based on national socioeconomic survey data.


62. National exam scores also improved, indicating better student performance.26 Although average scores in Bali and NTB were lower than the national average in 2001, they were higher than that average in 2012. NTT’s average exam scores were lower than the national average in 2012 (Table 6).

Table 6: National Exam Scores in Selected Provinces

Province Average (2001)

Average (2012)

Percentage Increase

Bali 6.48 8.06 24.38

Nusa Tenggara Barat 6.32 7.87 24.52

Nusa Tenggara Timur … 6.83 …

Indonesia 6.54 7.48 14.37

Sources: Study team’s calculations based on data from Indonesian Family and Life Survey and Ministry of National Education.

63. Improvements in enrollment, primary completion, junior secondary transition, and exam scores were observed in selected project districts after the project was implemented. Tables 7 and 8 present the education outcomes in 56 surveyed project schools located in six districts before and after project implementation. These improvements were observed both for poor students as a group and for all students. According to stakeholder interviews, the project addressed constraints on access to basic education, such as the poor condition of basic infrastructure and school facilities that largely prevented children from going to school or held down their national exam scores.

26 For primary school, the national examination consists of 3 subjects: mathematics, Indonesian language,

and science. Each has a maximum score of 10, hence a student’s maximum score on the national examination will be 30. For junior secondary school, there are 4 subjects, namely mathematics, Indonesian language, science, and English (maximum of 40). Prior to 2003, the subjects examined at national level were 5 for primary school (moral education, Indonesian language, mathematics, social science, and science), and 6 for secondary school (moral education, Indonesian language, mathematics, social science, science, and English).


Table 7: Changes in Education Outcomes before and after Project Implementation for All Students in Selected Districts

Province Selected Districts

Enrollment (District Average: children/year/school)

Completion (District Average: children/year)

Transition (District Average: children/year)

National Exam Score (District Average: children/year)

Before DBEP After DBEP Before DBEP After DBEP Before DBEP After DBEP Before DBEP After DBEP Bali Bangli 30 32 38 43 26 29 6.85 6.78 Bali Buleleng 39 46 42 52 34 38 6.7 7.14 Nusa Tenggara Barat West Lombok 50 45 43 51 34 36 6.46 7.28 Nusa Tenggara Barat East Lombok 45 45 50 56 38 39 6.54 7.23 Nusa Tenggara Barat Mataram 49 49 63 55 45 47 6.84 6.54 Nusa Tenggara Timur

Timor Tengah Utara 43 44 26 55 11 33 6.4 7.32

DBEP = Decentralized Basic Education Project. Note: Enrollment estimates refer to the average number of children enrolled in a primary school in the district in the given periods. Completion estimates are for the primary level, while transition estimates are for junior secondary level. For the national exam score, the average score all of the subjects covered under the examination was used, with 10 being the maximum score. See footnote 26 for the description of Indonesia’s national exam. Source: Study team’s calculations based on a survey done by the team for 56 randomly selected project schools in the six districts.

Table 8: Changes in Education Outcomes before and after Project Implementation for Poor Students in Selected Districts

Province Selected Districts

Enrollment (District Average: children/year/school)

Completion (District Average: children/year)

Transition (District Average: children/year)

National Exam Score (District Average: children/year)

Before DBEP After DBEP Before DBEP After DBEP Before DBEP After DBEP Before DBEP After DBEP Bali Bangli 26 26 30 25 23 24 6.88 6.70 Bali Buleleng 23 26 25 30 19 21 5.95 6.84 Nusa Tenggara Barat West Lombok 24 28 23 31 16 21 5.93 6.82 Nusa Tenggara Barat

East Lombok

23 28 23 26 17 21 6.28 7.07 Nusa Tenggara Barat Mataram 35 36 25 25 25 25 6.63 6.70 Nusa Tenggara Timur

Timor Tengah Utara

26 29 17 30 9 28 6.98 7.23 DBEP = Decentralized Basic Education Project. Note: Enrollment estimates refer to the average number of children enrolled in a primary school in the district in the given periods. Completion estimates are for the primary level, while transition estimates are for junior secondary level. For the national exam score, the average score all of the subjects covered under the examination was used, with 10 being the maximum score. See footnote 26 for the description of Indonesia’s national exam. Source: Study team’s calculations based on a survey done by the team for 56 randomly selected project schools in the six districts.


Specifically, there is sufficient evidence that average education outcomes in the selected districts were significantly higher after the project was implemented than before it was carried out

64. A paired t-test analysis was made to determine whether the changes in the education outcomes of the surveyed districts were statistically significant. The statistical analysis was used to determine whether the mean education outcomes improved after the project was implemented. It was carried out using data from 56 project schools selected using both probabilistic and non-probabilistic approaches to ensure that each school in each province had an equal opportunity to be selected as part of the sample. The impact of the project on education outcomes was examined in the selected schools, doing so for all students and for poor students as a group.27 65. The findings suggest that the project had an impact on education outcomes for all students in those districts examined (Table 9). Specifically, there is sufficient evidence that average education outcomes in the selected districts were significantly higher after the project was implemented than before it was carried out.28 Increased enrollment indicates that students had greater access to basic education, while increased transition and completion indicate greater opportunities for these students to finish basic education. Improved performance can also be observed in a rise in national exam scores. Among school participation outcomes, the highest average difference was observed on completion, followed by transition.

Table 9: Paired t-test for All Students at District Level

Variable

Average Before DBEP

Average After DBEP Difference

Standard Error

t-statistic

Probability (T<t)

Enrollment 41.75 43.29 -1.54a 0.44 -3.52 0.0004 Completion 42.18 51.54 -9.36a 1.55 -6.03 0.0000 Transition 30.03 36.79 -6.76a 1.06 -6.39 0.0000 National exam score 6.65 7.04 -0.39a 0.06 -6.48 0.0000

DBEP = Decentralized Basic Education Project. Notes: H0: outcomes before DBEP = 0; Ha: outcomes after DBEP <0; degrees of freedom = 55. a Estimates are statistically highly significant at 1% significance level. Source: Study team’s calculations based on a survey done by the team for 56 randomly selected project schools in six districts: Bangli, Buleleng, East Lombok, Mataram, Timor Tengah Utara, and West Lombok. 66. The findings of the paired t-test analysis also show that the project helped increase access of the poor to basic education in the selected districts (Table 10). The increase in transition indicates greater opportunities for these poor students to complete basic education. The project also helped improve the performance of poor students in the national exam. Improvements in transition and completion were most notable among the enhanced education outcomes for poor students.

Table 10: Paired t-test for Poor Students at District Level

Variable

Average Before DBEP

Average After DBEP Difference

Standard Error

t- statistic

Probability (T<t)

Enrollment 26.45 28.75 -2.30a 0.19 -12.23 0.0000 Completion 23.92 28.06 -4.14a 0.81 -5.11 0.0000 Transition 18.20 23.50 -5.30a 0.92 -5.73 0.0000 National exam score 6.45 6.88 -0.43a 0.06 -7.41 0.0000

DBEP = Decentralized Basic Education Project. Note: H0: outcomes before DBEP = 0; Ha: outcomes after DBEP< 0; degrees of freedom = 55. a Estimates are highly statistically significant at 1% significance level. Source: Study team’s calculations based on a survey done by the team for 56 randomly selected project schools in six districts: Bangli, Buleleng, East Lombok, Mataram, Timor Tengah Utara, and West Lombok.

27 During the mission, the study team gathered data on education outcomes in the 56 selected schools for

2000–2013. 28 The statistical analysis was conducted using STATA version 13.1.


67. School development. The project developed the schools’ capacity in determining priority education challenges and identifying corresponding strategies to address them. It provided school officials with training on how to formulate school development plans, involve communities in preparation of the plans, and monitor progress of the schools’ development. The interviewed principals noted that some ADB consultants were easily accessible for consultation and queries even beyond working hours. Given these contributions, the project effectively paved the way for implementation of school-based management in Bali, NTB, and NTT. Just 2 years into project implementation, such management was made official national policy in 2003. 68. The project has also effectively changed the mind-set of both school- and district-level officials in implementing school-based management. The project’s condition requiring schools to develop school development plans before they could access block grants bolstered the capacity and decision-making skills at school level. School development plans have become the most important consideration when selecting priority school programs, according to the interviews. School development plans examined by the evaluation team were deemed comprehensive. Their content covered the identified education problems, corresponding strategies to address these problems, work schedule, budget, and monitoring schedule. 69. The flexibility in the use of the block grants helped enhance the capacity of schools in basic education management. Such flexibility allowed project schools to determine priority needs and design innovative activities to address them, as interviews with stakeholders revealed. In contrast, such other transfers from the central government to schools as School Operational Assistance Program (BOS) grants employ more rigid regulations regarding those activities they can be used to fund. 70. While the project’s block grants were mostly used for rehabilitation of school facilities, some schools adopted innovative approaches for school improvement. Anecdotal evidence gathered by the evaluation team reveals that a school in Anjani, East Lombok used the block grant to address malnutrition among its students, 30.0% of whom were found to be malnourished following medical checkups. The school therefore used the block grant in conducting a lunch-feeding program to address this problem. Some schools in Bangli, Bali used their block grants to send teachers to remote communities. Since these communities had no schools and children had to walk several miles to reach the nearest school, these schools used the grants to dispatch teachers who went to and taught children in these remote areas. According to the World Bank’s school-based management national survey in Indonesia from 2010, budget allocation is one of those areas where school principals perceived they have high autonomy in decision making. On average, 90.0% of principals said they had decision-making authority in allocating the school budget, recruiting teachers, setting the school vision and curriculum, and selecting textbooks and teaching materials.29 71. The project was also effective in inducing communities to participate actively in school development through the formation of school committees. School committees usually are comprised of community leaders, local entrepreneurs, teachers, parents, and heads of villages or districts. According to the World Bank survey, parents account for roughly 75.0% of school committee members, followed by council representatives at 20.0% and teachers at 4.0%. The representation of the stakeholders is even more

29 The World Bank school-based management national survey used data from a nationally representative

survey of about 400 public primary schools in Indonesia. The World Bank commissioned the RAND Corporation to conduct the survey. See Karam, K., Marshall, J., and Vernez, G. 2012. Implementation of School-based Management in Indonesia. RAND Corporation.

While the project’s block

grants were mostly used for rehabilitation of school facilities,

some schools adopted

innovative approaches for

school improvement


varied, as parents also serve in such other functions as religious leaders or representatives of nongovernment organizations (footnote 29). 72. School committees were actively engaged in the formulation and subsequent revisions, if necessary, of school development plans. Although schools were not required to report on achievements against their school development plans, according to interviews with stakeholders, school committees, along with principals, local government officials, and ADB staff and consultants, contributed to the formulation and subsequent revisions, if necessary, of the plans. 73. Local ownership of the project as fostered by community participation in school development also increased the communities’ commitment to contribute to the project voluntarily. Through the school committees, community members contributed labor and other in-kind assistance to support school projects. Stakeholders also deemed community involvement as having a significant effect on education quality. With the introduction of BOS, however, schools have been expected to eliminate tuition fees or other compulsory payments from parents.30 To some extent, this might have inhibited some parents who would like to assist the school financially. Nevertheless, the central government still permits schools to receive voluntary contributions from community members for school development. 74. District basic education development. Interviewed stakeholders reported that district education offices’ primary role under the project dealt with the provision of training, particularly in teacher development and evaluation of school development plans. The district education offices were much involved in evaluating school development plans and accordingly provided feedback to schools. Monitoring duties by supervisors and district staff also included providing feedback on principal and teacher performance, checking on the conditions of school facilities, monitoring or observing classrooms and instruction, assessing teacher training needs, and reviewing and approving the school budget (footnote 29). 75. The District Education Development Fund had been used to finance activities related to the district education officers’ capacity building such as trainings or seminars. The fund had also been used to assist teachers and school principals in improving the quality of education. Its support for training teachers, school principals, and supervisors included contextual teaching and learning, in-service training, competency-based training, mastery of teaching and learning materials, evaluation, and classroom management, among others. The training was provided to both project and non-project schools in a district, expanding capacity in both. Interviews with stakeholders revealed that district education boards are still in place as they are mandated under Ministerial Regulation No. 044/2002. As noted in the PCR, however, a new regulation on the management of education (PP 17/2010) made funding of the district education boards by districts discretionary and reduced the number of board members. 76. The project’s impact on capacitating district education offices to produce 5-year district education development plans and 5-year annual action plans was less notable than that of other outputs. As noted in the PCR, the district education development plans were supposed to be revised annually, but this was not done in all districts. Instead, districts base their annual implementation on the government’s annual work plan, in line with government procedures. Moreover, only a few districts

30 World Bank. 2010. Supporting the BOS Program. Education Update. Issue No. 01. Jakarta


integrate school development plans into their annual work plans. In 2012, just 12.0% of districts consolidated school development plans for use in their district education planning process.31 Despite this, those district officials interviewed said the training they received in formulating development plans proved useful, as district education offices are required by law since 2004 to create development plans. 77. Challenges at the district level on decentralizing basic education have been observed. One of these—high staff turnover at the district level—has hampered the project’s capacity building efforts. Local district leaders, including mayors or heads of district education offices, have full authority to move officers in various units or departments, including principals and district education staff. In some districts, there was insufficient knowledge transfer. However, in some districts in which the project was supported by a team of district education staff, the remaining staff members were able to keep up with progress. 78. Second, some districts have too many teachers and others too few since teacher deployment and mobilization have also come under the authority of every district government. Moreover, education appointments are perceived to be politicized in that they are made by district heads. According to interviewees, some staff appointments are based on political ties. 79. Third, district governments have yet to maximize their education budgets and a huge portion is spent on personnel. District governments allocate some 30%–40% of their budgets to education. Because around 85% of their education budgets is spent on personnel costs, limited funds are left for education development.32 80. Fourth, there are flaws in information systems that project districts use to track and publish statistics. Data-keeping has been a problem, as documents are discarded or deleted after 5 years due to storage issues. Stakeholders reported that no soft copies of data are available.

31 The figure is based on a World Bank survey of 50 Indonesian districts on the quality of local education

governance. See World Bank. 2013. Local Governance and Education Performance: A Survey of the Quality of Local Education Governance in 50 Indonesian Districts. Jakarta.

32 Ministry of Education and Culture. 2013. Overview of the Education Sector in Indonesia 2012: Achievements and Challenges. Jakarta.


D. Efficiency 81. The project is rated efficient. Despite the limited amount of grants, the project achieved notable results in enhancing local capacity for education governance and improving school infrastructure. Especially considering that the grants provided under the project were spread over a 3-year period, some stakeholders noted these to be quite small. Interviews with Ministry of National Education, school, and district officials revealed that some Rp20 million to Rp25 million is usually required to rehabilitate one classroom in an elementary school (measuring 8 x 7 meters) and Rp30 million to Rp35 million for one classroom in a junior secondary school (measuring 9 x 7 meters). Since a block grant amounted to around Rp90 million rupiah, three to four classrooms could be rehabilitated. The PCR nevertheless pointed out that junior secondary school facilities improved, in particular within NTT, with 80% of junior secondary schools having administrative offices and 60% having libraries and science laboratories. Moreover, community support helped lower the costs of rehabilitating facilities, as some parents or school committee members offered in-kind assistance such as free labor. In addition to contributing to school infrastructure, the project made huge contributions in improving schools’ and districts’ capacity for local education delivery. 82. Indeed, it is impossible for the project to address all the education-related infrastructure requirements in Bali, NTB, and NTT. Continued rehabilitation of school facilities is imperative, and particularly for those in impoverished areas such as a school in the city of Kefamenanu, district of Timor Tenga Utara, in NTT, which the evaluation team visited and found to have some classrooms that were badly damaged (Figure 6). As stated, the government has recently scaled up its efforts to rehabilitate schools in the country.

Figure 6: Condition of Classrooms in Timor Tengah Utara, Nusa Tenggara Timur, 2014

Source: Study team, taken during the evaluation mission.

Despite the limited amount of grants, the project achieved notable results in enhancing local capacity for education governance and improving school infrastructure


83. The PCR pointed out that progress under the loan was initially slow, but the pace did pick up throughout execution. On the other hand, the grant project required two extensions to achieve project objectives. The grant’s shorter implementation period of only 2.5 years proved inadequate for severely poor and often inaccessible districts. As noted in the PCR, funds were generally disbursed on time and the full budget allocation, as well as most of the additional funds made available by the rising exchange rate, was used each year. There had been an 11.0% increase from the appraisal amount of the project.33 The increase was actually 29.6% of the base cost of the project, the largest share of which was attributed to the payment for international consultants that was 127.0% of the appraisal amount. The person-months of the consultants, on the other hand, increased by only 12.8% (from 78 person-months at appraisal to an actual 88 person-months). 84. The external auditor found a few problems with procurement. Most of those problems identified related to overpayment for workshops and training. Under the loan component, 57 audit cases—relating mostly to fund misuse due to overpayment, unclaimed penalties, and undeposited bank interest—were identified. Under the grant component, 18 cases were identified. All 75 identified cases (involving less than 1.5% of schools) were resolved. E. Sustainability 85. The project is rated likely sustainable. Its contributions to improving capacity for local education governance are reinforced by continued transfer of knowledge between school and district officials who received training and their counterparts who were not beneficiaries of the project. According to interviews, teachers from project schools often serve as trainers in a forum called Teacher Networks or MGMP (Musyawarah Guru Mata Pelajaran). MGMP is a non-structural organization established in accordance with 1991 guidelines issued by the Directorate General of Primary and Secondary Education, and it seeks to promote teacher professionalism. 86. Moreover, the project’s spillover effects are particularly notable in school development plan formulation, which has been assimilated even by non-project schools and has become standard practice among schools in the country. Continued community involvement in school development through the school committees also constitutes an important aspect of the project outputs’ sustainability. School committees remain part of the formulation process of school activity and budget plans, which are similar to school development plans. School committees also take part in decision making on the use of BOS grants, which, according to interviews with stakeholders, must be based on agreement between the BOS management team in the school, teacher’s council, and school committee. 87. In contrast to the PCR’s finding that some stakeholders had deemed the project activities unsustainable without project support, certain activities funded by the block grants are still being implemented. In particular, support for poor students through scholarships, transport allowances, remedial programs, catch-up programs, and tutoring for students missing school are still being provided by some schools.34 The central and district governments provide the bulk of funding to support continued

33 In the PCR (Appendix 4), the subtotal of the actual foreign base cost was reported to be $23.399 million.

When the evaluation team recalculated the reported table, however, the actual foreign base cost was supposed to have been $23.401 million. This $2,000 difference was equal to Rp 18,840,000.

34 The study team surveyed 56 selected schools. Some 64% indicated that activities for poor students are still being implemented even after project completion.

The project’s spillover effects are particularly

notable in school development plan

formulation, which has been

assimilated even by non-project

schools and has become standard

practice among schools in the

country


35%

31%

17%

13%

4%Ministry of Education

District government

Donor agencies

Local communities and parents

Other sources

implementation of activities targeting poor students (Figure 7). District governments, donor agencies, or local communities also provide funding to such initiatives.

Figure 7: Project Schools’ Sources of Funding for Continued Assistance to Poor Students (%)

Source: Study team’s calculations based on a survey of 56 schools.

88. The maintenance and rehabilitation of school facilities does not appear to be a priority of schools. Primary schools use about 8.0% of BOS grants for the maintenance of school facilities compared to 19.0% for examination activities and 13.0% for teacher honoraria. BOS grants can be utilized only for facility maintenance and small repairs such as painting or repairing a leaky roof.35 As already mentioned, however, the central government recently scaled up its efforts to rehabilitate school infrastructure.

35 World Bank. 2011. Independent Monitoring Report: BOS Program Implementation 2008–2009. Jakarta

(World Bank Office in Indonesia).

CHAPTER 4

Other Assessment A. Impact 89. The project’s impact is rated significant. As discussed in Chapter 3, improved education outcomes have been observed in the provinces of Bali, NTB, and NTT. Improvements in enrollment, transition, completion, and exam scores indicate students’ enhanced access to and greater opportunities to complete basic education, as well as their better learning performance. 90. Moreover, the project made pioneering contributions in improving local education governance capacity. At the start of project implementation in 2001, very few school officials were knowledgeable about decentralization of basic education, according to interviews with stakeholders. At the time, the project was the only available platform where schools were invited to discuss and suggest their needs to the government. Moreover, the project’s contributions to enhancing local capacity for decentralized basic education have been extended to non-project schools and districts. 91. Most importantly, the project has influenced government policies related to decentralizing basic education. The project piloted the concept of school-based management in Bali, NTB, and NTT by improving schools’ capacity to formulate school development plans and encouraging community involvement in school development prior to enactment of the National Education System Law in 2003. That law stipulates the promotion of school-based management. Similarly, BOS partly adopted the project’s block grant system. Also, Regulation Number 101 for 2013 of the Ministry of Education and Culture (Chapter 3, section E, Number 3) states schools must provide school activity and budget plans, which are essentially similar to school development plans, for liquidation of the BOS. While those plans’ formats may different from that of school development plans, the project has helped develop school officials’ capacity in preparing such plans.

B. ADB and Executing Agency Performance 92. The Asian Development Bank’s (ADB) performance is rated less than satisfactory. High turnover of ADB’s project officers—seven in total—hampered effective and continued project supervision and management. Such frequent changes in ADB project officers also caused difficulties in communication between ADB and MONE. As the project was managed directly from ADB headquarters in Manila, the Indonesia Resident Mission’s role largely was to facilitate communication between MONE and ADB staff at headquarters. The resident mission’s responsibilities and contributions could have been maximized by tapping resident mission staff for project management and monitoring tasks.

Other Assessment 31

93. The performance of MONE, the executing agency, is rated satisfactory. MONE initially encountered difficulties in managing consultants and gathering data, but eventually it gained sufficient capacity as project implementation progressed. MONE reported facing challenges, particularly in coordinating with MORA, the project’s co-executing agency. Since MORA is one of five state ministries not decentralized under the law, structural misalignment limits the capacity of MONE and MORA to coordinate their functions relating to decentralized basic education. MORA’s provincial offices are largely responsible for madrasah, while MONE’s district-level officials are responsible to manage schools under their jurisdictions. MORA’s district offices can play only an operational role in implementing education policies developed by its central and provincial offices. 94. Coordination between MONE and MORA in involving provincial MORA officials in local educational development planning exercises is contingent on the attitude and relationships of individuals in the respective offices.36 Contact between madrasah and MONE’s district education offices is also generally limited. Moreover, while MONE is focused solely on the delivery of education services, MORA performs multiple religion-related functions and only oversees Islamic education through a single directorate general. As such, MORA has relatively less resources, expertise, and institutional structure to ensure the same level of government service provision for Islamic schools that general schools receive from MONE (footnote 36).

36 United States Agency for International Development. 2007. Analysis of the Current Situation of Islamic

Formal Junior Secondary Education in Indonesia. Washington, DC.

Since MORA is one of five state ministries not decentralized under the law, structural misalignment limits the capacity of MONE and MORA to coordinate their functions relating to decentralized basic education

CHAPTER 5

Issues, Lessons, and Follow-up Actions A. Issues 95. The project sought to promote decentralization of basic education in Indonesia primarily by improving local capacity at the school and district levels for basic education management. This evaluation found that the implementation of decentralized basic education in Indonesia continues to face challenges. 96. First, coordinating the functions and authority over basic education services between local and central governments is an area of concern. District governments now wield the greatest authority in basic education, as interviews with stakeholders revealed. How to coordinate central and district governments’ authority and responsibilities in managing basic education services is a concern. This problem is clearly highlighted in decentralization’s impact on teacher policy and student mobility. Although decentralization has made it easier for teachers to be redeployed within a district, transferring teachers between districts has reportedly become more difficult after decentralization. 37 Even if one district has many teachers and a neighboring district lacks teachers, teachers cannot be moved because they are under district authority. Similarly, student mobility is hampered. Students cannot move from one district to another, because each district usually has a quota for students coming from other districts. 97. Second, the election of district officials in Indonesia has somewhat affected the decentralization of basic education services. Although electing district officials supports the participation of local communities in democratic processes and ensures they have a voice in decision making, interviews with stakeholders highlighted that some district officials try to drum up support at election time through promises of free education. Given the notion of free education, some community members are reluctant to contribute toward school development. Furthermore, interviews with stakeholders revealed that some district officials appoint staff based on political ties, including of those involved in delivering basic education. 98. Third, interviews with stakeholders revealed that schools appear to be more compliant with the district governments than the central government. Although district authorities are now responsible for managing basic education services, the central government maintains certain responsibilities in decentralized basic education management, particularly in monitoring the delivery and quality of education services. 37 J. Tobias et al. 2014. Toward Improving Education Quality: Indonesia’s Promising Path. Working Papers in

Economics and Development Studies No. 201412. Bandung: Department of Economics, Padjadjaran University.

Issues, Lessons, and Follow-up Actions 33

Prior to decentralization, the central government sent local governments questionnaires, which the latter would send to schools. Today, it is often much more difficult for the central government to request data from schools, such as on school profile. The central government through MONE has established the online school database Dapodik (Data Pokok Pendidikan), but it is still not easy to obtain data from schools. B. Lessons

99. The project provides some important lessons for promoting decentralized basic education management in Indonesia. The PCR identified key lessons, including the importance of (i) flexible management by ADB and the government, coupled with good communication from the project team that enabled the project to maintain its rationale and objectives despite changing legal and administrative environments; (ii) systematic record keeping, given the poor documentation and recording of data on project activities, outputs, and outcomes at the central and district levels; (iii) improving data quality and the use of data in planning and management in districts; (iv) clearly explaining project objectives to stakeholders, as the project was primarily regarded as a funding mechanism and the goal of district capacity development was deemphasized; and (v) integrating project funds and activities into government processes, which could have further improved district capacity building. 100. In addition to these, this evaluation has identified the following lessons. One, decentralization of basic education can be bolstered through strong collaboration between MORA and MONE. To address the structural misalignment between MONE and MORA, the role of provincial governments in decentralized basic education should be strengthened, particularly in monitoring and evaluation. The Indonesian government appears to be moving toward intensifying the delegation of authority to provincial governments in basic education. From 2014, provincial governments have full authority in preparing the materials for the primary school final examination and oversee the district government’s implementation of this examination, as stipulated in Ministerial Regulation No. 102/2013 and another regulation of the head of the MONE Research and Development Center. 101. Two, good technological infrastructure at the school and district levels can help address concerns about data management. MONE has implemented the Dapodik online school database system. Schools are required regularly to submit data to this system. However, many schools do not have access to the internet to accomplish this task. School staff must also be capacitated to utilize this online database effectively. 102. Three, effective capacity building efforts are contingent on consistent supervision and support. High staff turnover among ADB’s project officers and district governments tended to hamper effective communication and rapport-building among stakeholders. Consequently, this disrupted efforts to promote decentralized basic education management. Means of effective transfer and sharing of knowledge through, for instance, continued training are imperative. For ADB’s part, increasing the role of the Indonesia Resident Mission to include project management and monitoring responsibilities could have helped address project implementation issues. As discussed earlier, the project was managed directly from ADB’s headquarters in Manila. Moreover, ADB could have exerted greater efforts in disseminating knowledge products to further promote the sharing of lessons learned and best practices in decentralized basic management to other Indonesian provinces or countries similarly endeavoring to decentralize basic education delivery and management.


C. Follow-up Actions 103. The evaluation proposes the three follow-up actions for ADB to pursue given its continuing support for education-oriented projects in Indonesia. Effective decentralization of basic education largely hinges on collaboration between MONE and MORA, which needs to be improved. MORA remains centralized, while MONE has already been decentralized. ADB should support measures to improve the coordination between the two ministries. When implementing education interventions, ADB should help strengthen the role of provincial governments in decentralized basic education, particularly in monitoring and evaluation. Sharing more information regarding the best practices of national education management between the two ministries can help improve their coordination. 104. ADB’s efforts to further improve access to basic education in Indonesia should be complemented with continued investments in enhancing transport and technological infrastructure and school facilities. Improving transport infrastructure such as roads, and particularly in remote areas, would allow more students to continue their studies to junior secondary schools, which are normally located in the subdistrict capital city. The technological infrastructure of schools should also be enhanced to improve their access to and utilization of the online Dapodik system so that school data are regularly updated. Moreover, support is necessary for continued school rehabilitation work, including in such remote areas as those in Bali, NTB, and NTT. ADB should align its education interventions with programs by the government or other development partners in these areas. 105. If ADB’s Human and Social Development Division of the Southeast Asia Department will examine whether future ADB support is still required toward decentralization of basic education in Indonesia, it should consider the challenges which the decentralized system continues to face. These include the impact of decentralization on teacher policy and perceived politicization of appointments made by district governments in the education area.

Appendixes

APPENDIX 1: SUMMARY DESIGN AND MONITORING FRAMEWORK

Design Summary Performance Indicators and Targets PCR/PPER Update Sustainability 1. Impact To support the government’s goal of achieving universal basic education that serves the needs of poor and marginalized people and is responsive to local needs

Improved indicators for gross enrollment rates of 120.7% for primary education and 78.9% for junior secondary education Improved human development indicators in targeted provinces and districts

Gross Enrollment Rate, by Province and School Type Province 2010 2011 2012 2013

Primary Bali 95.53 90.39 91.06 94.28 Nusa Tenggara Barat 95.16 92.69 94.56 96.63 Nusa Tenggara Timur 93.03 92.13 92.28 93.60 Indonesia 94.76 91.03 92.49 95.53

Junior Secondary Bali 67.83 69.16 75.07 80.69 Nusa Tenggara Barat 71.73 76.70 77.81 80.18 Nusa Tenggara Timur 51.03 56.74 55.89 59.24 Indonesia 67.73 68.12 70.84 73.72

Source: Indonesian Statistics Office. Human Development Indicators, by Province

Province 2010 2011 2012 Bali 72.28 72.84 73.49 Nusa Tenggara Barat 65.20 66.23 66.89 Nusa Tenggara Timur 67.26 67.75 68.28 Indonesia 72.27 72.77 73.29

Source: Indonesian Statistics Office.

While targets for gross enrollment rate may not have been achieved, the impact is sustainable, especially considering the long-term effects of these improvements on education outcomes in terms of, for instance, employability.

2. Outcome Poor children’s participation, transition, completion, and performance in basic education in Bali, Nusa Tengarra Barat (NTB), and Nusa Tengarra Timor (NTT) improved

In each project district: increased net enrollment rates; improved primary completion rates and junior secondary graduation rates; improved primary to junior secondary transition rates; and improved 6th- and 9th-grade national examination scores

Net Enrollment Rate, by Province and School Type Province 2010 2011 2012 2013

Primary Bali 95.53 90.39 91.06 94.28 Nusa Tenggara Barat 95.16 92.69 93.56 96.63 Nusa Tenggara Timur 93.03 92.13 92.28 93.60 Indonesia 94.76 91.03 92.49 95.53

Junior Secondary

Bali 67.83 69.16 75.07 80.69 Nusa Tenggara Barat 71.73 76.70 77.81 80.18 Nusa Tenggara Timur 51.03 56.74 55.89 59.24 Indonesia 67.73 68.12 70.84 73.72

Source: Indonesian Statistics Office. Completion Rates, by Province and School Type

Province 2008 2009 2010 2011 Primary

Bali 98.28 99.01 99.26 99.49

Improvements in education outcomes among the poor are expected to have long-term impacts, for instance on employability and inequality.

Sum

mary D

esign and Monitoring Fram

ework 37

Design Summary Performance Indicators and Targets PCR/PPER Update Sustainability Nusa Tenggara Barat 99.16 99.91 99.20 98.89 Nusa Tenggara Timur 98.88 98.00 98.46 98.72 Indonesia 99.18 98.97 98.84 99.05

Junior Secondary Bali 93.12 99.07 99.27 98.90 Nusa Tenggara Barat 95.60 96.94 97.79 96.81 Nusa Tenggara Timur 95.37 95.98 98.00 97.73 Indonesia 95.66 97.01 97.50 96.96

Source: Study team’s calculations based on national socioeconomic survey data. National Exam Scores, by Province

Province Average (2000)

Average (2012)

Bali 6.48 8.06 Nusa Tenggara Barat 6.32 7.87 Nusa Tenggara Timur … 6.83 Indonesia 6.54 7.48

Source: Study team’s calculations based on national socioeconomic survey data.

Among children from the poorest 40% of families: increased net enrollment rates; improved primary completion rates and junior secondary graduation rates; improved primary to junior secondary transition rates; and improved 6th- and 9th-grade national examination scores

Net Enrollment Rate for Poorest 40% of Students, by Province and School Type

Province 2000 2003 2006 Primary

Bali 92 92 93 Nusa Tenggara Barat 89 92 94 Nusa Tenggara Timur 89 88 92 Indonesia 92 93 94

Junior Secondary Bali 58 54 51 Nusa Tenggara Barat 53 51 66 Nusa Tenggara Timur 27 34 41 Indonesia 51 55 59

Source: ADB. 2012. Completion Report: Indonesia: Decentralized Basic Education Project. Manila. Examination Scores and Transition Rates for Poor Students in NTT, by School Year and School type

05/06 06/07 07/08 08/09 Examination Score

Primary 6.0 6.0 6.0 5.9 Junior Secondary 5.7 5.7 5.0 5.7

38 Appendix 1

Design Summary Performance Indicators and Targets PCR/PPER Update Sustainability Transition Rate

Primary 94 96 96 97 Junior Secondary 88 91 87 94

Source: ADB. 2012. Completion Report: Indonesia: Decentralized Basic Education Project. Manila.

School-based management (SBM) implemented in project schools

SBM implemented in about 3,000 primary and junior secondary schools and madrasah in project districts Project schools meet minimum conditions for learning Effective community–school partnerships

SBM was implemented in 5,240 project schools. School development plans (SDPs) were developed by all project schools. Small rehabilitation grants were distributed to 3,522 schools (67%) and quality improvement grants to all schools. There was high community involvement in schools, leading to stronger school–community partnerships.

SBM is now a national policy supported by the National Education Law (20/2003).

Districts in Bali, NTB, and NTT manage decentralized basic education effectively

Increased funding for basic education over 2001 levels Data-based and goal-oriented 5-year development plans for basic education developed Plans for equity and quality improvement in project districts implemented Districts meet goals for achieving equitable access to basic education

National Education Law (2003) requires national and local governments to allocate at least 20% of their budgets, thereby considerably increasing district funding for education since 2001. All districts developed district education development plans under the project, although the use of data was limited. Some districts now produce annual plans in line with district budget processes rather than district education development plans. Plans for equity and quality improvement in districts were implemented.

Districts are mainly focused on monitoring and evaluating schools’ improvement plans. There is weak capacity at the district level for gathering and updating data.

3. Outputs 3.1. School-based management implemented in schools School baseline analysis and needs assessment carried out and school development plans (SDPs) developed SBM implemented Community involvement

All SDPs contain baseline analysis and needs assessment, realistic targets and strategies for school improvement. SBM implemented in schools through: functioning and active school committees; integrated school budgets; increased decision making at school level; annual school staff performance

Participatory baseline studies and needs assessments were carried out before SDPs were developed. All schools established school committees and held regular meetings. In all provinces, 95% of stakeholders interviewed had attended meetings. Community contributions were mainly in the form of labor, while monetary contributions were made by only 35%–40% of the school community in NTB and Bali and 25% in NTT.

SBM is now a national policy supported by the National Education Law (20/2003). The formation of school committees is also stipulated under this law. Moreover, the practice of SDP formulation has been assimilated even by non-project schools.

Sum

mary D

esign and Monitoring Fram

ework 39

Design Summary Performance Indicators and Targets PCR/PPER Update Sustainability in schools strengthened

review; and school-initiated innovations for improving effectiveness

3.2 School quality improved with School Development Fund assistance Improved school effectiveness Increased number of poor children completing basic education More schools meet minimum conditions of learning

Project schools meet SDP targets for (i) education outcomes including enrollment, completion, transition, exam scores, and teacher discipline; and (ii) classroom improvement, including classroom number, teaching aids, and other facilities and materials

See education outcomes above. The attendance of teachers and students increased steadily every year in the NTT districts, from an average of 83%–88% to 90%–95% (not measured in loan districts). For NTB and Bali: 90% of stakeholders reported that the conditions of the schools, classrooms, furniture, and toilets improved. There was less improvement in libraries and laboratories (only a requirement for junior secondary schools). At the end of the project, 71% of classrooms were in good condition and furniture was available in 60%–70% of schools. Availability of teaching and learning materials improved moderately, but the needs remain large. For NTT: only 60% of primary and 47% of junior secondary schools classrooms were in good condition at the end of the project, but almost all schools had adequate furniture. The proportion of primary schools with libraries increased from 23% to 40%, and junior secondary schools with laboratories from 50% to 60%. Only 39% of toilets were in good condition by the end of the project. Schools received teaching and learning materials (generally textbooks) and science kits, but the need for other kinds of materials remains high.

Given knowledge sharing and transfer among school officials, training and capacity building efforts are expected to be more sustainable. Improvements in school facilities such as classrooms are less evident and sustainable, however, inasmuch as the block grants provided may be too small to accommodate rehabilitation and maintenance of these facilities.

3.3. District capacity to plan and manage basic education increased District education leadership strengthened 5-year district basic education development plan completed Information systems in place

Education boards established with community representation, according to decree Project districts revise 5-year plans regularly on the basis of up-to-date information Timely and accurate district annual statistics produced

Education boards in all districts received training during the project, including training in budget analysis. The boards work in partnership with the district education offices to improve district education while reporting directly to the head of the district government. The major work of the boards revolves around monitoring and evaluation of education programs and education financing (School Operational Assistance Program and national special allocation fund); capacity building and supervision of school committees; and mediation among parents, schools, and district education offices. All districts prepared education development plans during the

High turnover of staff at the district level poses challenges on proper transfer of knowledge and continued capacity building in basic education management.

40 Appendix 1

Design Summary Performance Indicators and Targets PCR/PPER Update Sustainability project, but not all districts have continued the practice after the project. Annual education statistics are produced in districts, albeit with a time lag (sometimes of several years). The failure of schools to submit information is blamed. In addition, capacity is lacking in this area in many district education offices.

3.4. District education improved with District Education Development Fund assistance Management of basic education provision strengthened

In district education offices, systems in place for planning, managing, financing, staffing, and monitoring of: teacher development; school supervision; principal recruitment, training, and performance review; facilities expansion and improvement; and district financing of education

All districts have generally developed systematic approaches in all these areas, with the weakest being teacher development. The effectiveness of the Decentralized Basic Education Project’s programs in building the capacity of district staff was acknowledged. All districts reported that project-trained staff are highly regarded and keenly sought after to fill positions.

High staff turnover at the district level poses challenges for proper transfer of knowledge and continued capacity building in basic education management.

3.5. Effective program and financial compliance monitoring Independent and internal monitoring of financial management and program procedures

The School Development Fund and District Education Development Fund are managed in compliance with government and Asian Development Bank requirements.

Financial compliance systems were successful. Only a small number of cases of funds misuse were identified and all were resolved.

Proper budgeting and monitoring is part of training provided under SDP and the project’s formulation.

Note: Presentation of updates to indicators cited in the project completion report was made depending on data availability. Sources: Study team’s calculations using Indonesian Statistics Office data; ADB. 2012. Completion Report: Indonesia: Decentralized Basic Education Project. Manila.

APPENDIX 2: LIST OF INTERVIEWEES A.1 Ministry of Education

1. Mr. Hamid Muhammad, Director General, Directorate General Office for Basic Education (formerly Manager of the Decentralized Basic Education Project [DBEP])

2. Mr. Didik Suhardi, Director, Junior Secondary Education (formerly a member of the DBEP Central Technical Committee)

3. Mr. Dedi Karyana, Head, Section for Education Infrastructure (formerly a member of the DBEP Central Technical Committee)

4. Mr. Budi Susetyo, Asian Development Bank consultant A.2 District Education Offices Bali Province Buleleng District

1. Mr. Ketut Arjana, Head, Basic Education Affairs, District of Buleleng (formerly DBEP District Manager)

2. Ms. Ni Ketut Santrini, Section Head, Basic Education Affairs District of Buleleng (formerly DBEP Treasurer, District of Buleleng)

Bangli District

1. Mr. I Nyoman Sumantra, Head of the District Education Office, District of Bangli 2. Mr. Wayan Sudiarsa, Secretary of the District Education Office, District of Bangli (formerly DBEP

Manager, Bangli) 3. Mr. I Made Artana Yasa, Head of Finance, District Education Office, District of Bangli (formerly

DBEP Manager, Bangli) 4. Mr. Ketut Punia, Staff, District Education Office, District of Bangli (formerly DBEP officer)

West Nusa Tenggara Province Lombok Barat (West Lombok) District

1. Mr. Iswarta, Section Head for Teacher and Administration Staff, District of West Lombok (formerly DBEP District Manager)

2. Ms. Dini, District Education Officer, West Lombok Lombok Timur (East Lombok) District

1. Mr. Supriyadi, Head, Basic Education Affairs, District of East Lombok (formerly DBEP District Manager)

2. Mr. Abdul Muin, Officer, District of East Lombok (formerly DBEP District officer)

Mataram City

1. Mr. Zaenal Arifin, Head of the Basic Education Affairs, District Education Office, Mataram City

42 Appendix 2

East Nusa Tenggara Province Timor Tengah Utara District

1. Mr. Emanuel Anunu, District Crops Plantation Officer, former DBEP project manager 2. Mr. Thimateus Timo, District Education Officer, former DBEP officer

A.3 Project Schools1 Bali Province Buleleng District

1. Mr. Pande Ketut Remawa (Principal, Junior Secondary School #3 KBT) 2. Mr. Made Imawan (Principal, Junior Secondary School #1 Tejakula) 3. Mr. Gede Suastika (Principal, Junior Secondary School #45 Sukasada) 4. Mr. Gede Sumatra Jaya (Principal, Junior Secondary School #4 Sukasada) 5. Mr. Nyoman Purnayasa (Principal, Junior Secondary School #2 Singaraja) 6. Mr. I Made Sutarjana (Principal, Junior Secondary School #1 Seririt) 7. Mr. Nyoman Suyasa (Principal, Junior Secondary School #2 Seririt) 8. Mr. Ketut Suardika (Principal, Junior Secondary School #1 Sawan) 9. Mr. I Wayan Kantun Toni (Principal, Primary School #1 Astina) 10. Ms. Gusti N. Reniasih (Principal, Primary School #1 Br Tegal) 11. Ms. Ketut Sudarmini (Principal, Primary School #5 Br Jawa) 12. Ms. Ni Wayan Resmi (Teacher, Primary School #1 Br Bali) 13. Mr. Made Ketu Raja (Principal, Primary School #2 Br Tegal) 14. Mr. Nyoman Sudiarsa (Principal, Junior Secondary School #5 Singaraja)

Bangli District

1. Mr. I Putu Jati (Principal, Primary School #3 Peninjoan) 2. Mr. I Nengah Sudana (Principal, Primary School #2 Abuan) 3. Mr. I Nyoman Wardana (Principal, Primary School #6 Tiga) 4. Mr. I Nengah Suara (Teacher, Primary School #3 Tiga) 5. Mr. I Ketut Disna (Teacher, Junior Secondary School #2 Bangli) 6. Mr. I Nengah Sulatra (Principal, Junior Secondary School #2 Kintamani) 7. Mr. Wayan Bukahan (Principal, Primary School #2 Selat) 8. Ms. Dayu KM Suastini (Teacher, Primary School #2 Selat) 9. Mr. I Nengah Genah (Principal, Primary School #4 Taman Bali)

West Nusa Tenggara Province Lombok Barat (West Lombok) District

1. Mr. Syahrudin Saat (Principal, Primary School #7 Lambuak) 2. Ms. Raodah (Principal, Junior Secondary School #4 Gerung) 3. Mr. Abdul Haris (Principal, Junior Secondary School #1 Lempau) 4. Mr. Akhmad S. (Principal, Primary School #1 Kediri)

1 The study team collected information from representatives of 56 project schools. Some principals and teachers were not able

personally to participate in the interviews held at the district education offices but were given questionnaires, which they completed and returned to the study team.

List of Interviewees 43

5. Mr. Abdul Gani (Principal, Junior Secondary School #1 Gunung Sari) 6. Mr. Apudin (Principal, Primary School #4 Gerung Utara) 7. Mr. Saringin (Principal, Junior Secondary School #3 Lingsar) 8. Mr. M Rusman (Principal, Primary School #1 Gelogor) 9. Mr. Tajudin (Principal, Junior Secondary School #5 Lingsar) 10. Ms. Sabariah (Principal, Primary School #1 Labuapi)

Lombok Timur (East Lombok) District

1. Mr. Agus Gunarto (Vice Principal, Junior Secondary School #1 Suralaga) 2. Mr. Abdul Mantif (Principal, Primary School #1 Anjani) 3. Ms. Hj. Seripa Fatimah (Principal, Primary School #2 Sandubaya) 4. Mr. M. Tohran (Principal, Primary School #1 Dusun Lekong) 5. Mr. Bambang Subagyo (Principal, Junior Secondary School #1 Masbagik) 6. Mr. Nursalim, Section Head for Curriculum Affairs, District of East Lombok (formerly DBEP

school principal, Primary School #3 Sehket)

East Nusa Tenggara Province Timor Tengah Utara District

1. Ms. Matilda K Bano (Principal, Primary School Baki) 2. Mr. Petrus Obe (Principal, Junior Secondary School Napan) 3. Mr. Fransiskus Meko (Teacher, Primary School Nunpene) 4. Mr. Marselinus Kolise (Principal, Junior Secondary School Biboki Utara) 5. Mr. Benekdiktus Falo (Principal, Primary School Koko) 6. Mr. Dominikus Tasoeb (Principal, Christian Secondary School St. Ga) 7. Mr. Benekdiktus Seo (Principal, Christian Secondary School St. Paulus) 8. Mr. Alexander Pilis (Principal, Secondary School Oeluele) 9. Mr. Yoseph Sila (Principal, Secondary School Teflasi) 10. Mr. Raymundus Fallo (Principal, Junior Secondary School #2 Kefa) 11. Mr. Nikolas Neonnub (Principal, Junior Secondary School #2 Muotire) 12. Mr. Gustaf Maf (Vice Principal, Christian Secondary School Kefa)

APPENDIX 3: QUESTIONNAIRE FOR SCHOOLS

KUESIONER SEKOLAH

Informasi Responden

Nama Responden:

Nama Sekolah:

Posisi di Sekolah:

Alamat Sekolah:

Nomor Telepon:

Sekolah anda terpilih sebagai salah satu penerima bantuan dari proyek ADB, Desentralisasi Pendidikan Dasar (DBEP)

Pada kuesioner ini, kami ingin menanyakan hal-hal yang terkait dengan pelaksanaan proyek tersebut

Pada khususnya, kami ingin mengetahui bagaimana sekolah ini mengambil manfaat dari proyek tersebut

Bagaimana proyek ini membantu sekolah dalam mengimplementasikan rencana pengembangan sekolah

Bagaimana proyek ini membantu aktifitas sekolah dan bagaimana dana dari proyek ini dialokasikan ke berbagai aktifitas tersebut

PERTANYAAN 1: Berapa total dana ADB yang diterima oleh sekolah antara tahun 2001-2010, apabila ada? 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

Kami akan bertanya mengenai berbagai kegiatan di sekolah Anda yang dibayai oleh dana ADB antara periode 2001 dan 2010.

PERTANYAAN 2: Berapa besar bantuan dari ADB yang digunakan untuk berbagai aktivitas berikut? 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

i. Bantuan sekolah kepada siswa miskin

ii. Program alternatif untuk siswa yang tidak dapat mengikuti kelas reguler

iii. Segala jenis perbaikan fasilitas sekolah

iv. Pembangunan ruang kelas

v. Pembangunan fasilitas air bersih dan sanitasi

vi. Pembelian buku ajar dan buku lainnya untuk perpustakaan

vii. Penyediaan alat bantu ajar

viii. Peningkatan kapasitas guru

ix. Aktifitas lainnya, sebutkan…..

Q

uestionnaires for School 45

Kami akan bertanya mengenai dukungan sekolah untuk siswa miskin

Tujuan kami menanyakan hal ini adalah untuk mengetahui proporsi dari siswa yang mendapatkan manfaat dari dana ADB dan besaran bantuan yang mereka terima.

Pertanyaan 3: Seberapa besar dana ADB digunakan untuk berbagai kegiatan berikut 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

diantara tahun 2001 dan 2010?

i. Beasiswa

ii. Bantuan uang transport

iii. Program remedial

iv. Kegiatan kelas akselerasi

v. Kelas tambahan bagi siswa yang absen di kelas

vi. Bentuk bantuan lainnya, sebutkan…….

PERTANYAAN 4: Berapa banyak siswa di sekolah Anda yang 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

menerima bantuan sesuai dengan kegiatan yang tercantum di pertanyaan 3

PERTANYAAN 5: Berapa banyak siswa miskin di sekolah Anda 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

yang menerima bantuan sesuai dengan kegiatan yang tercantum di pertanyaan 3?

Kami akan bertanya mengenai proses penentuan berbagai aktivitas di sekolah Anda.

PERTANYAAN 6 : Dari berbagai aktivitas yang disebutkan diatas, faktor apakah yang menjadi pertimbangan untuk menentukan suatu kegiatan?

Pilih faktor berikut

i. Rencana pembangunan sekolah

ii. Permintaan orang tua siswa

iii. Permintaan dari dinas pendidikan setempat

iv. Permintaan dari guru

v. Permintaan dari kepala sekolah

vi. Permintaan dari siswa

vii. Permintaan dari lembaga donor

viii. Lainnya, sebutkan….

46 Appendix 3

PERTANYAAN 7: Siapa yang menginiasi pembuatan dokumen perencanan sekolah? Pilih jawaban berikut

i. Kepala Sekolah / Wakil Kepala Sekolah

ii. Komite Sekolah

iii. Pemerintah lokal

iv. Donor (ADB)

v. Lainnya, sebutkan…

PERTANYAAN 8: Siapa yang terlibat dalam pembuatan dokumen perencanaan sekolah? Pilih jawaban berikut

i. Kepala Sekolah / Wakil Kepala Sekolah

ii. Komite Sekolah

iii. Pemerintah lokal

iv. Donor (ADB)

v. Lainnya, sebutkan…

PERTANYAAN 9: Apakah dokumen perencanaan sekolah ini pernah dievaluasi oleh Dinas Pendidikan

i. Ya (ke pertanyaan 10)

ii. Tidak

PERTANYAAN 10: Apakah rekomendasi yang diberikan oleh Dinas Pendidikan bermanfaat?

i. Ya

ii. Tidak

Q


Kami ingin bertanya apakah kegiatan yang pernah dibiayai oleh proyek ini masih berlangsung (sustainabel)

PERTANYAAN 11: Apakah aktifitas pada pertanyaan 3 di atas masih dilakukan setelah proyek selesai ?

i. Ya (Ke pertanyaan 12)

ii. Tidak

PERTANYAAN 12: Kementerian (Dinas) dan/atau lembaga swasta mana yang membantu membiayai aktifitas tersebut?

Pilih jawaban berikut

i. Kementerian Pendidikan dan Kebudayaan

ii. Dinas Pendidikan

iii. Lembaga Donor, yaitu ….

iv. Komunitas lokal atau orang tua siswa

v. Lainnya , sebutkan …

Sekarang kami akan bertanya mengenai bagaimana siswa mengambil manfaat dari proyek ini dalam hal tingkat pendidikannya

PERTANYAAN 13: Apakah proyek ini meningkatkan pendaftar ke sekolah ini? Mohon berikan jumlah pendaftar tahun 2000 - 2013

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

PERTANYAAN 14: Apakah proyek ini meningkatkan jumlah siswa yang diterima di jenjang yang lebih tinggi (SD ke SMP, atau SMP ke SMA)? Mohon berikan data tersebut dari 2000 - 2013.

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

48 Appendix 3

PERTANYAAN 15: Apakah proyek ini meningkatkan tingkat kelulusan di sekolah ini? Mohon berikan data siswa yang berhasil lulus dari tahun 2000–2013.

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

PERTANYAAN 16: Apakah proyek ini meningkatkan nilai Ujian Nasional (untuk siswa kelas 6 atau kelas 9) di sekolah ini? Mohon berikan data rata-rata nilai UN tersebut dari 2000 –2013

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

PERTANYAAN 17: Apakah proyek ini meningkatkan pendaftar dari siswa miskin di sekolah anda? Berikan data siswa miskin yang mendaftarkan diri tahun 2000 –2013

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

PERTANYAAN 18: Apakah proyek ini meningkatkan jumlah siswa miskin yang diterima di jenjang yang lebih tinggi (SD ke SMP, atau SMP ke SMA)? Mohon berikan data tersebut dari 2000 –2013.

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

PERTANYAAN 19: Apakah proyek ini meningkatkan tingkat kelulusan siswa miskin di sekolah ini? Mohon berikan data siswa miskin yang berhasil lulus dari tahun 2000 –2013.

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

PERTANYAAN 20: Apakah proyek ini meningkatkan nilai Ujian Nasional siswa miskin (kelas 6 atau kelas 9) di sekolah ini? Mohon berikan data rata-rata nilai UN siswa miskin tersebut dari 2000–2013

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Q


Kami ingin meminta pendapat mengenai kesuksesan proyek ini

PERTANYAAN 21: Bagaimana proyek ini memberi manfaat untuk sekolah ini? Apakah proyek ini meningkatkan kualitas pendidikan di sekolah ini?

PERTANYAAN 22: Apakah anda menginginkan sekolah anda untuk terus menerima bantuan ADB di masa yang akan datang?

Note: The English version of this questionnaire will be available upon request.

APPENDIX 4: NUMBER AND TYPE OF PROJECT SCHOOLS

Table A4: Number and Types of Schools Covered by the Decentralized Basic Education Project

Item Number of

Districts

Primary School

Primary Madrasah

Junior Secondary School

Junior Secondary Madrasah

Total Public Private Public Private Public Private Public Private Loan Bali 9 715 0 15 16 114 58 5 3 926 Nusa Tenggara Barat 9 1,946 1 55 319 303 35 8 167 2,834 Nusa Tenggara Timur 2 308 94 0 7 57 16 0 0 482 Subtotal 20 2,969 95 70 342 474 109 13 170 4,242 Grant Nusa Tenggara Timur 6 468 363 3 19 96 44 1 2 996 Total 26 3,437 458 73 361 570 153 14 172 5,238 Source: ADB. 2012. Completion Report: Indonesia: Decentralized Basic Education Project. Manila.

APPENDIX 5: RATING MATRIX FOR CORE EVALUATION CRITERIA

Table A5.1: Rating Matrix for Core Evaluation Criteria Ratings PCR PPER Reasons for Disagreements/Comments Relevance Highly

Relevant Highly

Relevant The Decentralized Basic Education Project (DBEP) was implemented

during the start of the decentralization era in Indonesia and helped to fill gaps in technical capacity.

The project aimed to improve access to and quality of basic education, especially for the poor.

The project is aligned with the priorities of ADB and the Indonesian government.

Effectiveness Effective Effective Based on questionnaires sent to schools and mean-difference testing, beneficiaries on average had seen statistically significant improvements in terms of students’ enrollment, transition, completion, and exam scores. a This is also true for poor students.

The project made a significant contribution to lowering dropout rates of poor children in such poor districts as Bangli, East Lombok, and Timor Tengah Utara, either by sending teachers to remote areas or providing facilities for poor children.

High turnover of trained officers at the district level may have hindered the project’s effectiveness. For some districts in which DBEP personnel were organized in teams, however, the remaining staff were able to keep up with the turnover.

Efficiency Efficient Efficient Despite the limited size of grants, the project achieved notable results in improving local capacity for basic education delivery and rehabilitating some school infrastructure.

Sustainability Partly Likely Some of the project’s outputs are still implemented under current government programs, such as creating the proposal for a program or project similar to school development plans even though the format of such reports may not be exactly the same.

School committees, which were pioneered by the project, are still functioning.

It was difficult, however, to determine the project’s impact on the rehabilitation of school facilities since other sources of funding for school repairs and rehabilitation have been made available after the project’s life.

Impact Significant Significant Though difficult to measure because of the differences in method and sampling frame used between baseline survey (2006, 2008) and the study team’s analysis, DBEP was found to have a significant impact on school-based management and community involvement.

The project had a positive spillover impact or peer effect on nonbeneficiary schools. As some training funded by the District Education Development Fund was attended by all teachers and principals of both beneficiary and nonbeneficiary schools, DBEP had an effective impact on changing the behavior of teachers and principals toward school-based management and on improving education quality in their schools.

Institutional Development

Not Rated Significant The project had made a significant contribution toward institutional development, in particular through the establishment of school committees and education boards at district level. Community involvement had also improved.

a With an exception of the District of West Lombok, where the average student enrollment for the selected schools decreased after DBEP was implemented.

Source: Study team’s assessment.

52 Appendix 5

Table A5.2: Core Evaluation Criteria Rating Values

Rating Value Relevance Effectiveness Efficiency Sustainability 3 Highly relevant Highly effective Highly efficient Most likely 2 Relevant Effective Efficient Likely 1 Less than relevant Less than

effective Less than efficient

Less likely

0 Irrelevant Ineffective Inefficient Unlikely Source: Independent Evaluation Department. 2006. Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. Manila: ADB (as well as its amendment effective from March 2013).

APPENDIX 6: EDUCATION STATISTICS

Table A6.1: Net Enrollment in Primary School by Beneficiary Provinces, 2003–2013 (%)

Province 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Bali 91.58 93.48 93.26 93.33 94.49 94.93 94.99 95.53 90.39 91.06 94.28 Nusa Tenggara Barat 92.48 92.42 92.99 94.50 94.20 94.20 94.75 95.16 92.69 93.56 96.63 Nusa Tenggara Timur 88.27 90.79 92.00 91.58 91.61 91.72 92.46 93.03 92.13 92.28 93.60 Indonesia 92.55 93.04 93.25 93.54 93.78 93.99 94.37 94.76 91.03 92.49 95.53

Note: Since 2007, the net enrollment rate includes nonformal education (“Package A,” which is primary school; “Package B,” which is junior secondary). Source: Indonesian Statistics Office.

Table A6.2: Net Enrollment in Junior Secondary School by Beneficiary Provinces, 2003–2013

(%) Province 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Bali 68.63 69.37 70.03 70.15 66.69 67.34 67.38 67.83 69.16 75.07 80.69 Nusa Tenggara Barat 57.19 61.70 67.53 69.62 70.79 71.44 71.32 71.73 76.70 77.81 80.18 Nusa Tenggara Timur 39.10 43.26 43.00 47.23 49.75 49.87 50.21 51.03 56.74 55.89 59.24 Indonesia 63.49 65.24 65.37 66.52 66.90 67.39 67.43 67.73 68.12 70.84 73.72

Note: Since 2007 the net enrollment rate includes nonformal education (“Package A,” which is primary school; “Package B,” which is junior secondary). Source: Indonesian Statistics Office.

Table A6.3: Classroom Conditions by Province, 2002

(%)

Province

Primary School Junior Secondary School Heavily

Damaged Lightly

Damaged Good Heavily

Damaged Lightly

Damaged Good DKI Jakarta 4.48 14.64 80.88 2.26 9.76 87.98 Jawa Barat 32.40 36.45 31.15 5.20 10.92 83.88 Banten 25.74 28.25 46.00 4.93 10.59 84.48 Jawa Tengah 18.94 38.37 42.69 1.49 7.76 90.74 DI Yogyakarta 14.13 40.70 45.17 2.38 8.40 89.21 Jawa Timur 17.55 36.87 45.58 1.96 6.37 91.67 N Aceh Darussalam 23.25 34.91 41.84 4.21 10.93 84.86 Sumatera Utara 20.22 38.64 41.14 3.74 9.65 86.61 Sumatera Barat 18.24 38.88 42.89 4.47 10.99 84.54 Riau 30.54 28.62 40.84 0.92 4.20 94.88 Jambi 22.26 25.61 52.12 3.48 9.14 87.38 Sumatera Selatan 21.74 33.17 45.09 1.90 6.50 91.60 Bangka Belitung 13.26 29.93 56.81 1.95 6.56 91.50 Bengkulu 31.56 35.66 32.78 6.61 14.08 79.32 Lampung 28.70 45.60 25.70 2.86 8.83 88.31 Kalimantan Barat 28.39 33.32 38.29 2.87 9.67 87.46 Kalimantan Tengah 30.92 31.47 37.61 1.83 3.48 94.69 Kalimantan Selatan 27.52 33.18 39.30 3.40 11.01 85.58 Kalimantan Timur 22.31 36.00 41.70 3.42 9.29 87.29

54 Appendix 6

Province

Primary School Junior Secondary School Heavily

Damaged Lightly

Damaged Good Heavily

Damaged Lightly

Damaged Good Sulawesi Utara 17.16 33.04 49.79 7.23 17.40 75.37 Gorontalo 21.86 32.95 45.19 7.54 12.87 79.59 Sulawesi Tengah 34.03 33.00 32.97 3.53 6.85 89.62 Sulawesi Selatan 19.69 31.52 48.78 4.08 11.08 84.83 Sulawesi Tenggara 31.99 33.12 34.89 3.74 9.01 87.25 Maluku 36.24 30.98 32.78 9.57 13.74 76.69 Maluku Utara 38.07 29.50 32.43 9.46 18.72 71.82 Bali 20.08 35.33 44.59 4.79 10.33 84.88 Nusa Tenggara Barat 16.64 32.97 50.39 1.68 9.63 88.69 Nusa Tenggara Timur 36.68 29.37 33.96 7.40 11.94 80.66 Papua 23.42 25.49 51.09 9.47 8.38 82.15 National Average 24.27 32.92 42.82 4.28 9.94 85.78 Source: National Development Planning Agency. 2002. National Program for Indonesian Children: Education. Jakarta.

APPENDIX 7: POVERTY STATISTICS

Table A7: Percentage of Poor People by Province, 2007–2012

Province 2007 2008 2009 2010 2011 2012 Papua 37.08 37.53 36.80 31.98 31.11 30.66 Papua Barat 35.12 35.71 34.88 31.92 28.20 27.04 Maluku 29.66 28.23 27.74 23.00 21.78 20.76 Nusa Tenggara Timur 25.65 23.31 23.03 21.23 20.88 20.41 Gorontalo 24.88 25.01 23.19 18.75 17.33 17.22 Nusa Tenggara Barat 23.81 22.78 21.55 19.73 18.63 18.02 Aceh 23.53 21.80 20.98 19.57 19.46 18.58 Lampung 20.98 20.22 18.94 16.93 16.18 15.65 Sulawesi Tengah 20.75 18.98 18.07 15.83 15.40 14.94 Bengkulu 20.64 18.59 18.30 17.50 17.70 17.51 Sulawesi Tenggara 19.53 18.93 17.05 14.56 13.71 13.06 Jawa Tengah 19.23 17.72 16.56 15.76 15.34 14.98 Jawa Timur 18.51 16.68 15.26 14.23 13.40 13.08 DI Yogyakarta 18.32 17.23 16.83 16.08 16.05 15.88 Sumatera Selatan 17.73 16.28 15.47 14.24 13.78 13.48 Sulawesi Barat 16.73 15.29 13.58 13.89 13.24 13.01 Sulawesi Selatan 13.34 12.31 11.60 10.29 10.11 9.82 Jawa Barat 13.01 11.96 11.27 10.65 10.09 9.89 Sumatera Utara 12.55 11.51 11.31 11.33 10.67 10.41 Maluku Utara 11.28 10.36 9.42 9.18 8.47 8.06 Kalimantan Barat 11.07 9.30 9.02 8.60 8.17 7.96 Sumatera Barat 10.67 9.54 9.50 9.04 8.19 8.00 Riau 10.63 9.48 8.65 8.47 8.22 8.05 Sulawesi Utara 10.10 9.79 9.10 8.51 8.18 7.64 Kalimantan Timur 9.51 7.73 7.66 6.77 6.68 6.38 Jambi 9.32 8.77 8.34 8.65 8.42 8.28 Kepulauan Riau 9.18 8.27 8.05 7.40 7.11 6.83 Kalimantan Tengah 8.71 7.02 6.77 6.56 6.51 6.19 Kepulauan Bangka Belitung 8.58 7.46 6.51 5.75 5.53 5.37 Banten 8.15 7.64 7.16 6.32 5.85 5.71 Kalimantan selatan 6.48 5.12 5.21 5.29 5.06 5.01 Bali 6.17 5.13 4.88 4.20 4.18 3.95 DKI Jakarta 4.29 3.62 3.48 3.75 3.69 3.70 Indonesia 15.42 14.15 13.33 12.49 11.96 11.66 Source: Indonesian Statistics Office. 2014.

Indonesia: Decentralized Basic Education Project · the decentralized basic education sector were...

Documents

Transcript of Indonesia: Decentralized Basic Education Project · the decentralized basic education sector were...