1

Impact Evaluation of the Mediation and Community Legal Empowerment component

of the Support for Poor and Disadvantaged Areas project, Indonesia

Concept Note

1 October 2008 Introduction The Government of Indonesia and the World Bank are planning to rigorously monitor and evaluate the Mediation and Community Legal Empowerment (MCLE) component of the Support for Poor and Disadvantaged Areas (SPADA) project. SPADA is one of the largest World Bank projects in Indonesia. The main objective of the SPADA program is to help the Indonesian government in addressing the problems of governance and poverty in the poorest districts in the country. This project is also intended to help conflict affected provinces return to normalcy. The MCLE component targets districts that are particularly conflict prone.

A draft comprehensive monitoring and evaluation framework has been developed.1 This framework includes the following key components:

Ongoing monitoring of key management and performance indicators Process evaluation of initial implementation Impact evaluation, including qualitative and quantitative baselines and endlines Cost benefit and effectiveness evaluation Special studies, e.g. gender review, complaint handling

This concept note outlines the impact evaluation component. It includes information on the MCLE program, the impact evaluation’s research themes and questions, indicators, methods, quantitative and qualitative research tools, methodological and analytical outputs, dissemination, research phases and timelines, and the impact evaluation team. Motivation This impact evaluation provides a unique and exciting opportunity to examine the welfare impacts of providing access to justice to poor communities. This evaluation will be the first instance of rigorous quantitative and qualitative evidence on the effect of the provision of community level legal services and education on a range of human rights and economic outcomes. This unique opportunity drives two main motivations for this impact evaluation. The first reason is more global, and relates to the fact that the evaluation would help to build a more 1 See the draft document, Monitoring and Evaluation Framework for the Mediation and Community Legal Empowerment component of the Support for Poor and Disadvantaged Areas program, November 2007, WB/Bappenas/KPDT.

2

rigorous empirical basis for undertaking justice sector reform across the developing world. The second is more local and specifically related to current policy and programming dynamics in Indonesia. Global Motivations Beginning to build a more rigorous empirical basis for analyzing and assessing the impact of our ongoing justice sector reform programs is crucial. While there is broad agreement on the importance of building—and enhancing access to—“rule of law” systems in developing countries, we have a much poorer understanding of how such goals should be realized and efforts to do either of these things have a fairly unhappy history. One of the reasons for these disappointments is arguably that reforms have lacked a sound empirical basis and that there is limited evidence of what does and does not work in a particular context. This dearth of systematic research remains despite the fact that it is 40 years since the law and development movement2 and almost 20 years since the revival of interest in justice sector reform in the area.3 Indeed, there continues to be little consensus on what a successful project entails and there are few examples where major impacts have been reported, despite the fact that the Bank and other donors continue to commit hundreds of millions of dollars to justice sector reform initiatives. Local Motivations In Indonesia, this impact evaluation is crucial for informing current policy debates on justice reform and access to justice as well as Government and Bank poverty, social justice and empowerment programming in Indonesia. The Government of Indonesia’s planning agency, Bappenas, is currently preparing its first ever Access to Justice Strategy for inclusion in the government’s 2010-14 Medium-Term Development Plan. The impact evaluation will inform the operationalization of this national plan as well as the development of future plans. Furthermore, the Bank is working with the government to establish PNPM, a nation-wide poverty and empowerment program. This new initiative includes a social justice component, which includes access to justice. Again, this impact evaluation will be critical to the scale-up of MCLE or similar access to justice programs in these national government initiatives. Program Background The MCLE program was designed based on a number of Government and World Bank analytical studies and pilot programs.4 The program’s main aim is to increase access to

2 For discussions about the ‘failure’ of the law and development movement, see Trubek and Galanter (1974); Merryman (1975); Burg (1975). 3 For a discussion of the problem of knowledge in rule-of-law programs, see Carothers (2004). 4 See, for example, Government of Indonesia (2002) Kecamatan Development Program: KDP in Conflict Areas, Jakarta: Ministry of Home Affairs, Directorate General of Rural and Community Empowerment, KDP National Secretariat and National Management Consultants (NMC); World Bank (2004) Village Justice in Indonesia: Case Studies on Access to Justice, Village Democracy & Governance, Jakarta: World Bank; and Patrick Barron, Rachael Diprose & Michael Woolcock (2006) Local Conflict and Community Development in Indonesia: Assessing the Impact of the Kecamatan Development Program, Working Paper No.10, Social Development,

3

justice for rural communities so that they can resolve conflicts and legal problems themselves. This can include access to both formal and informal justice systems.5 MCLE has a number of specific objectives:

1. Strengthen the capacity of institutions and individuals at community level to resolve disputes in an open, independent and just manner;

2. Assist and facilitate the resolution of disputes involving community-wide interests; 3. Increase public trust in the formal legal system by enhancing community legal

awareness and access to the formal system. To achieve these aims and objectives, MCLE establishes a support network for providing legal aid and empowerment activities. The main focus of activities is at community level. MCLE establishes a network of Community Legal Aid Posts (Posko) at the sub-district-level. The posts will function as a centre for legal information, mediation and complaints handling. The Posko will be staffed by a full-time facilitator (one person, fasko, per Posko) and paralegals in selected villages (two per village – one woman and one man). They will work in partnership with local leaders and community figures, including religious leaders, the village head and adat institutions. They will also work with judges, prosecutors and police. The Community Lawyer at the district-level will provide technical legal support on a part-time pro bono basis. The table below summarizes the program locations in the two provinces. Province District # Kecamatan /

Posko (total) # Villages (total) # Paralegals

Aceh Aceh Barat 4 (9) 32 (282) 64 Aceh Besar 4 (22) 32 (595) 64 Aceh Utara 4 (21) 32 (1160) 64 Pidie 4 (30) 32 (946) 64 Maluku Maluku Tengah 4 (19) 32 (312) 64 Maluku Tenggara 4 (13) 32 (116) 64 Total: 2 6 24 192 384 Note: The figures in brackets for Pidie and Maluku Tengah indicate the recent district splitting. The main activities or interventions are:

Community legal education Capacity building for informal justice actors Legal aid and complaint handling network Awareness raising of community legal needs and challenges

Bappenas and the Ministry for Disadvantaged Areas (KDPT) are jointly managing the program. KPDT will contract a local organization in each province (university or NGO) to operate as the Provincial Implementing Agency for MCLE, with overall responsibility for World Bank Office Jakarta. Pilots on which the program is based include the Revitalisation of Legal Aid and Women’s Legal Empowerment programs, for more information see www.justiceforthepoor.org 5 Access to justice is defined as: “Access by people, in particular from poor and disadvantaged groups to fair, effective and accountable mechanisms for the protection of rights, control of abuse of power and resolution of conflicts. This access includes the ability of people to seek and obtain a remedy through formal and informal justice systems, and the ability to seek and exercise influence on law-making and law-implementing processes and institutions.” See Bedner (2004).

4

implementing the MCLE component. The SPADA NMC Community Legal Empowerment specialist will provide technical support to the Provincial Implementing Agencies. The program is funded from a grant in Aceh (DFID through the World Bank) and a loan in Maluku (SPADA). UNDP have allocated funding, to be implemented through the World Bank, to support implementation of the program in Aceh. Research Questions The central approach to this impact evaluation is to take a broad, multidimensional view on what welfare is and to trace the chain of effects stemming from the intervention. We will examine the effects of MCLE on knowledge of rights, exercise of rights, dispute resolution, reports and resolution of corruption, investment and household consumption. We expect that it will take longer for some of these effects to manifest than others and there are no clear experiences which tell us what this timeframe is. Thus, we complement the practical solution of conducting multiple follow up surveys with the conceptual approach of looking at these different links in the chain empirically to see how the program affects progress. The project team identified six key questions for the evaluation to answer.

1. Does MCLE increase beneficiaries’ ability to claim rights, enforce rights, and resolve disputes through non-violent means?

2. Does MCLE reduce violent conflict and increase safety and security? 3. Does MCLE improve household welfare in the villages that receive paralegals? 4. Does MCLE increase the reporting of corruption and bribery and decrease these

events in the long term? 5. Does MCLE improve individuals’ capacity to engage with the state and the state

capacity to respond, particularly in matters involving the resolution of disputes? 6. Does MCLE improve gender relations at the household and community level?

5

Indicators Note: this list of indicators is geared more towards the quantitative element of the impact evaluation. We discuss the qualitative approach in more depth below in the section on methods. However, it should be borne in mind that these indicators will also guide the qualitative exercise. No

Indicator Source of Information (see section Research Tools below)

Research Question 1: Impact on rights claiming and dispute processing 1 Changes in dispute processing preferences (actual and hypothetical)

- Percentage decrease in number/type of cases not reported (or preference for not reporting)

- Increased use of formal legal institutions

HH Survey KI Survey

2 Improvements in dispute processing practices - Percentage increase in number/type of cases successfully

resolved - Increased level of satisfaction with outcomes - Decrease in the number of steps required to resolve

disputes - Decrease in the time taken to resolve disputes - Decrease in the cost of resolving disputes - Increased collection of compensation agreements

HH Survey

3 Substantive improvements in dispute processing practices - Increased reference to and consideration of state law

(national as well as local) - Increased reference to and consideration of local adat norms

and claims - Decreased levels of discriminatory decision making - Increased adoption of humane sanctions - Increased acceptance of the right to appeal, and the right to

speak and be heard

Case Study Analysis

4 Increased assertion of rights and claims against authorities in beneficiary communities

- Increased willingness to voice individual concerns and raise community problems

- Increased assertion of legal identity and ownership rights (e.g., ID cards, birth certificates, land certificates)

- Increased assertion of community service and development rights (e.g., health services, education services, roads)

HH Survey C Survey

Research Question 2: Impacts on violence, safety and security 4 Decrease in the number of violent incidents in beneficiary

communities HH Survey C Survey Newspaper Monitoring

5 Decrease in the intensity of violent incidents in beneficiary communities

HH Survey C Survey Newspaper Monitoring

6 Increased perceptions of safety and security HH Survey

6

- Increased perceptions of personal safety - Increased perceptions of property security

Research Question 3: Impacts on welfare 1 Increased household consumption amongst beneficiary communities HH Survey 2 Increased household investment amongst beneficiary communities

(e.g., housing improvements, land investment and improvements HH Survey

3 Increased equality and fairness in the distribution of outcomes in cases involving economic/financial resources

Case Study Analysis

Research Question 4: Impacts on corruption and bribery 7 Decrease in the number of corruption/bribery incidents in beneficiary

communities HH Survey KI Survey

8 Changes in dispute processing preferences relating to corruption/bribery in beneficiary communities

- Percentage decrease in the acceptance of corruption/bribery

- Increased willingness to report corruption/bribery incidents (particularly involving development programs, such as SPADA)

- Percentage decrease in unreported incidents of corruption/bribery

HH Survey

9 Perceived improvements in the processing of corruption/bribery cases in beneficiary communities

- Perceived decrease in corruption/bribery due to deterrence - Increased perceptions that corruption/bribery cases would

be duly addressed and prosecuted by formal legal institutions

HH Survey KI Survey

10 Substantive improvements in the processing of corruption/bribery cases in beneficiary communities

- Increased use of formal legal institutions to process corruption/bribery cases

- Increased consistency in the interpretation and application of corruption/bribery laws

- Decreased levels of political manipulation of case outcomes - Increased satisfaction with outcomes of corruption/bribery

cases - Increased enforcement of corruption/bribery cases

Case Study Analysis

Research Question 5: Impacts on state-society engagement 11 Increased use of formal legal institutions for dispute processing (see

indicator 1 above) HH Survey KI Survey

12 Increased use of formal state institutions to protect legal identity and ownership rights (see indicator s 1 and 4 above)

HH Survey

13 Increased use of formal state institutions to protect collective service and development rights (see indicator 4 above)

C Survey

14 Increased perceptions of trust, fairness and efficiency of formal legal institutions

- Police (at various levels), Courts, Prosecutors Office, Lawyers, Land Office, Civil Registry, Camat, etc.

HH Survey KI Survey

15 Increased levels of community collective action - Increased participation in village decision making

HH Survey KI Survey

7

- Increased influence over village decision making - Increased willingness to challenge higher authorities - Increased frequency of community initiated projects - Increased associational activity

C Survey

Research Question 6: Impacts on gender relations 16 Increased gender equality in household decision making HH Survey 17 Increased female involvement in community collective action

- Increased female participation in village decision making (see indicator 15 above)

- Increased female influence over village decision making (see indicator 15 above)

HH Survey KI Survey

18 Substantive improvements in the processing of disputes involving the interests of female disputants

- Increased reporting of disputes involving women’s interests - Increased use and consideration of state law protecting

women’s interests (e.g., domestic violence legislation) - Decreased levels of discriminatory decision making along

gender lines (see indicator 3 above) - Increased acceptance of a female disputants right to appeal,

to speak, and be heard (see indicator 3 above)

Case Study Analysis

Evaluation Method This section discusses the methodological approach to the impact evaluation. The project is targeted at four levels (See Figure 1 below). First, two provinces, Aceh and Maluku, have been chosen because of the level of conflict prevalent there. Within each province, a number of districts or kabupaten (4 in Aceh, 2 in Maluku) have been chosen based on an assessment of the greatest need for the program and demand from local stakeholders. These cannot be changed. As a result, any impact evaluation will be generalizable only to areas of a similar context to the chosen kabupaten. Below the kabupaten (district) level, there are two levels of targeting where the program will be randomly allocated. First, there is the selection of kecamatan (sub-districts) where the posko or community legal post will be located. In Aceh, 4 out of 11-30 kecamatan per kabupaten will be chosen (for a total of 16 out of 82). In Maluku 4 out of 10-19 kecamatan per kabupaten will be chosen (for a total of 8 out of 29). Second, only a random selection of villages will receive paralegals. In Aceh, this is estimated to be about 8 of 30 villages per kecamatan, for a total of 128 of 728 villages. In Maluku, this will be 8 of approximately 10-15 villages per kecamatan, for a total of 64 of approximately 112 villages. The table in the program background section provides more details on numbers and locations. Figure 1 on the following page provides a graphical illustration of program assignment and impact evaluation design.

8

Figure 1: Program Assignment Design for SPADA MCLE Project (Aceh & Maluku) Admin Level

Stages of Randomization Stage A: Random selection process for sub-districts (kecamatan) receiving Poskos and for sub-districts

assigned as controls. Stage B: Random selection process for villages receiving paralegals. Treatment and Control Groups T1: Villages receiving posko and paralegal T2: Villages receiving only posko C: Control group with no posko and no paralegal T1 vs C: Effect of having access to a posko and paralegal versus nothing T2 vs C: Effect of having access to only a posko but no paralegal versus nothing T1 vs T2: Combined effect of posko and paralegal versus only posko NOTE: A third treatment group has been identified for a World Bank Conflict Resolution Training (CRT) Program. Villages for this treatment will be randomly selected through the same sampling process but will be separate from T1, T2 or C. The World Bank’s Conflict Team will use both this third treatment and C to conduct a separate evaluation.

Kabupaten/ Lawyer District (6 kabupaten) (Preselected)

Stage B: T1 T2 C Desa/ Paralegal No Paralegal No paralegal Village Total: 192 Total: ~1,000 (128 Aceh, 64 Maluku)

Stage A: Kecamatan/ Posko No Posko Sub District Total: 24 Total: 88 (16 in Aceh 8 in Maluku) (64 in Aceh and 24 in Maluku)

9

Randomization at the kecamatan level (as well as the village level) will help us answer the question: what would be the effect of the project if we were to make it available to all kecamatan in similar kabupatens? From the point of view of scaling up the intervention, this is the most relevant answer. By not explicitly selecting the kecamatan where this project is most likely to succeed, we can learn what the effects of this intervention would be in the average kecamatan. Indeed, by adding a good baseline on kecamatan and village institutions, we can increase our understanding of what are the critical aspects of these institutions that help the project have a bigger impact. The MCLE program design has two variations. One set of treatment villages, identified as T1 in Figure 1, receive Posko at the kecamatan level AND paralegals at the village level (192 villages). The other set of treatment villages (T2) are villages in kecamatan where there are Posko but the village does not receive paralegals (600 villages in Aceh and 48 villages in Maluku). The control group is drawn from villages in the same kabupaten but different kecamatan. These villages will have access to neither paralegals nor posko. Comparing T1 or T2 to the control group will allow us to measure the impact of both versions of the program relative to nothing. Comparing T1 to the control group would provide the combined effect of both the posko and the paralegal. Comparing T1 vs. T2 allows us to measure the effects of having a posko only relative to having both poskos and paralegals. This additional layer of comparison is useful because providing poskos only is clearly much cheaper. Evaluating the effect of poskos plus paralegals versus the control (T1 vs C) is the priority for this evaluation, with the comparison of posko and paralegal versus Posko only is conducted only in the cases where we have a sufficient sample of villages in the kecamatan that were selected. Note that in the 4 selected kabupaten in Aceh there are a total of 2700 villages making this analysis feasible in Aceh, but not in Maluku where there are far fewer villages in selected kecamatan. Threats to evaluation design Given the strategies outlined above, there are a number of potential threats to the validity of the evaluation analysis/results. These are particularly relevant to the quantitative tools. • The random kecamatan selection process will be subverted due to the influence of

kecamatan or kabupaten officials. This will introduce a difference in unobservable (as well as observable) characteristics between the treatment and control populations. This could bias the results. To avoid this risk, the selection process has to be clearly agreed by all parties beforehand, and closely monitored during implementation. To this end, the initial lists of selected kecamatan and villages have been prepared by the evaluation team in Washington D.C.

• In Maluku, the selection process for kecamatan may result in an unbalanced number of Christian and Muslim communities, resulting in pressure to alter the chosen communities and/or jeopardizing the sustainability of the project. To avoid this risk, it may be desirable to stratify the communities to be selected based on religion, before the selection rule is

10

applied. In the first instance, it is worth examining how many of these communities can be characterized as Christian or Muslim. Currently, no allowance for this has been made. The team will check to make sure this imbalance is not an issue before the program is rolled out.

• The main two programs that are likely to confound the effects of this project is the Women’s Legal Empowerment (WLE) project and Conflict and Resolution Training (CRT) program, also sponsored by the Bank in Aceh. The WLE project will operate in 13 kecamatan in 5 districts in Aceh. Three of the districts overlap with this project. There is no scope to affect this allocation. WLE choices should be known by the time the survey for MCLE starts, so this can be taken into account in the sampling size and strategy of the MCLE baseline. The CRT program will provide training to village leaders in selected villages across these districts and will also provide training to community facilitators. The program is being coordinated with MCLE. Villagers selected to receive the CRT program were identified through the same sampling process as MCLE and CRT will utilize the control areas from the MCLE sample to conduct a separate evaluation of CRT.

• A risk to the evaluation results arises from the fact that we are attempting to measure dispute resolution. This has two core problems. First, the measurement of dispute resolution in quantitative surveys to date has been rather weak, and knowledge of how to do this well is thin. To mitigate this problem, the evaluation team includes members of the World Bank Living Standards Measurement Study team who have extensive experience in designing and implementing surveys in developing countries. Members of the evaluation team have conducted extensive pretest of the survey instrument (attached as an annex) and fine tuned the questions (including discarding questions which did not work well). Second, the frequency of disputes may be sufficiently low so as to require either an unaffordable sample or yield insignificant results (i.e. the project had a real impact, but we will not be able to measure this accurately). To allay this risk the household survey will collect data on proximate indicators (e.g. hypothetical dispute resolution, individuals’ knowledge of their rights). In addition, the pre-testing has indicated that disputes are quite common in these areas – an average of more than 1 for the last two years in Aceh and over 2 for Maluku.

Research Tools This section summarizes both the quantitative and qualitative tools. The team will work together to ensure that the early results of both the quantitative and the qualitative work inform later data collection efforts using each method. Generally the quantitative research of the impact evaluation will focus on “does” questions whereas the qualitative research will focus on “how” questions. For example, the quantitative research will ask “does MCLE improve the welfare of beneficiaries?” whereas the qualitative research will ask “how does MCLE improve the welfare of beneficiaries?” Quantitative Research Tools The quantitative tools and data collection efforts will involve six tools: a household survey, key informant survey, community survey, program facilitator survey, newspaper conflict

11

monitoring, and MIS data. They are listed below in order of importance (conditional on adequate funds being available, they will all be pursued). The quantitative surveys will involve baselines and endlines before the project starts implementation. A second round of surveys (the mid-term survey) will take place 2 years after implementation is initiated in the first community. The project is currently scheduled to run until 2010, so these mid-term surveys may be seen as the end of project data (and so in the timeline at the end of this document they are referred to as endline surveys). However, the team feels that some of the effects will take longer to manifest (and the project may, in fact, be extended) and hence the current plan is to execute a third round of surveys four years after implementation. The surveys will be fielded during the same time each year to deal with problems of seasonality. The baseline data will be analyzed in consultation with the project team within 3 months of collection to provide for refinements in project design (e.g. identifying the most pressing legal issues in the communities). 1. Household Survey Baseline, midterm and endline household surveys will be administered. The questionnaire will contain a standard set of demographic and socioeconomic information (household composition, consumption expenditure, housing condition) consistent with the national household survey (Susenas). In addition, the questionnaire will contain questions specifically pertaining to outcome indicators asked of the head of household and/or his/her spouse (violence, crime, dispute resolution experiences, participation in village life, perceptions of safety and dispute resolution, and the like). The baseline survey will be a random sample of households in selected villages/communities. The follow-up surveys will be designed to measure changes in outcomes. A "partial panel" design will be considered, where the midterm/endline households are partially a panel of households interviewed in the baseline survey and partially a new cross-section to allow for migration/population shifts in the 2 year interval. In addition, prior to the midterm/endline survey, potentially tracking baseline households that moved out of the village/community will be considered/reviewed, since households who moved who might have differential outcomes (for example, if they moved due to a dispute that could not or was not resolved). 2. Key Informant Survey Baseline, midterm and endline key informant surveys will be administered to village authorities (including informal leaders). The questionnaire will contain demographic and socioeconomic information, as well as questions pertaining to outcome indicators at the individual and village-level (legal knowledge and opinions, perceptions of safety and security, corruption, service delivery, and claim-making). 3. Community Survey

12

Baseline, midterm and endline community surveys will be administered to village authorities and informal leaders using a group interview method. The questionnaire will contain questions about village characteristics (such as geographical accessibility, information accessibility, access to state institutions, etc.), the presence of other government and NGO programs, the frequency and type of both individual and community disputes in the village, and community claim-making. 4. Program Facilitator Survey A baseline (and possibly endline) survey of paralegals and posko staff will be administered to all program staff working at the village-level (paralegals) and kecamatan level (fasko). This survey will attempt to provide measures of attributes of these individuals that may prove important for project success (e.g. education levels, experience, social connections, legal knowledge). 5. Newspaper Conflict Monitoring As part of its monitoring and evaluation, the SPADA program will collect data on conflict and crime levels and impacts as reported by local newspapers in all nine SPADA provinces. This data source will be particularly relevant for examining the research questions relating to violent conflict levels, security and safety, and incidence of (reported) corruption. 6. MIS Data The impact evaluation will also utilize the monitoring data on the program functioning. This will include data on posko activities, paralegal activities, and case handling data. This will also possibly include a small satisfaction survey of posko and/or paralegal users. Qualitative Research Tools The impact evaluation will utilise three main tools to establish a qualitative baseline: village mapping, the collection of comparative case studies, and community ethnographies. 1. Village Mapping The qualitative research will involve undertaking an initial mapping of village dynamics. The aim of this exercise will be both to familiarise the researchers with their research villages, and to provide preliminary data in order to determine which case studies to investigate and research in-depth. The village mapping will occur in parallel to and in close cooperation with the quantitative survey enumerators. It will focus on documenting the following:

Basic Village Profile – The researchers will produce a basic village profile outlining demographics, livelihoods, poverty, location, etc.

Dispute timeline – Working closely with the quantitative enumerators the qualitative researchers will determine a two year dispute timeline for each village. This will establish the sequence of disputes, the type of disputes, the actors (disputants,

13

witnesses, mediators, etc.) and their relationships, and the issue in dispute. The timeline will distinguish between individual and community-wide disputes.

Claims and collective action timeline – Similar to the dispute timeline, the researchers will determine a two year claims and collective action timeline. Again the timeline will distinguish between individual and community claims.

Power relations and networks – The researchers will also map key village leaders and their relationships with others in the village and those external to the village, as well as the nature of their relationship.

2. Comparative Case Studies The qualitative research will involve collecting in-depth case studies in treatment and control locations. The main purpose of the case studies is to investigate the direct effects discussed above, but also examine indirect or spillover effects. The diagram below depicts the main processing pathways for disputes and claims in MCLE and non-MCLE villages. “Yes” and “No indicates whether the component was utilised. Notes: LLI: Lembaga Lingkungan Indonesia (Non-government Organization) This dispute pathway (tree) generates a number of possible pairs for conducting comparative case study research. For the purposes of this evaluation, the key comparisons are probably:

1. Effect of MCLE presence (D compared to J) 2. Effect of paralegal (C compared to D, and C compared to B) 3. Effect of posko (E compared to F, and E compared to B) 4. Effect of MCLE on LLI (G compared to I)

Given that the program will obviously not have started at the time of the baseline qualitative research, the baseline survey would focus on disputes and claims in cells D, G (but that jump

Impacts (HH)

Dispute and Claim Processing

Conflict, Safety and Security

Welfare

Corruption and Bribery

State-Society Relations

Social Cohesion Dispute/Claim

A. MCLE

C. Paralegal: Yes

E. Posko: Yes

G. LLI: Yes

H. LLI: No

F. Posko: No

D. Paralegal: No

B. Non-MCLE

I. LLI: Yes

J. LLI: No

14

straight from A), I and J. It would probably be most strategic, given limited resources, to focus on a restricted number of common dispute and claim types like, for example, land and divorce disputes, and land and alimony claims. The quantitative survey and the initial village mapping could assist with this type selection. 3. Community Ethnographies In addition to the qualitative case studies, the qualitative component of the impact evaluation will involve conducting in-depth ethnographies in a small number of program villages. The ethnographies will tend to focus on the indirect and spillover effects discussed above. At this state it is anticipated that two community (village) ethnographies will be conducted in each district (4 districts in Aceh and 2 districts in Maluku). The ethnographies will focus on how the program operates locally and its interaction with and effect on local dispute processing and power structures in the community. They will also track the experience of key individuals, including program staff (paralegals), potential and actual users, as well as community authorities and informal leaders. The ethnographies will be conducted over the course of program implementation by an anthropologist or anthropology PhD student. Outputs The impact evaluation will generate a number of methodological and analytical outputs. The box below lists those anticipated at the outset. Methodological Outputs Analytical Outputs 1. Quantitative Surveys:

a. Household Survey b. Key Informant Survey c. Community Survey d. Program Facilitator Survey e. User Satisfaction Survey

2. Quantitative Survey Field Guide 3. Qualitative Research Guide (Village Mapping

and Case Studies) 4. Qualitative Research Guide (Community

Ethnographies)

1. Baseline Analytical Report 2. Baseline Case Studies (Approximately 40

cases) 3. Thematic Case Study Compilations 4. Community Ethnographies (2 per district) 5. Endline and Midline Analytical Reports 6. Impact Evaluation Report (mixed methods) 7. Other analytical reports (as identified)

Dissemination Strategy Evaluation Design: The impact evaluation has been developed in consultation with a small number of government staff working on the program. However, to ensure wider understanding within the relevant program ministries (Bappenas and KPDT), the impact evaluation design is being presented to the government. The evaluation design will also be presented to the implementing partners in Aceh and Maluku once selected.

15

Baseline Survey Results: Data from the baseline surveys will be analyzed and the results presented to key stakeholders, including the government executing agencies, the NGO implementation partners, key program staff as well as internally within the Bank. The report and presentations will explain the targeting exercise, baseline community profiles, baseline levels of legal knowledge and understanding, disputes, dispute processing, and preferences, and the use of legal mechanisms. Impact Evaluation Results: The results of the impact evaluation (endline as well as midline if results are sufficient) will be disseminated to a number of stakeholders in a range of ways and forums. 1. Results from the impact evaluation should be presented in a workshop with government and NGO implementing partners directly involved in the program. The purpose of these workshops should be to:

• Illustrate the impact of the program • Show how findings can be used to modify the program • Examine the circumstances and groups for whom the program has the most impact

2. In addition to government departments and NGOs directly involved in the program, the results should also be disseminated to a broader range of government departments, local NGOs, and World Bank staff through a series of meetings, presentations (BBLs), and workshops. Particularly relevant are those involved in PNPM and the development, operationalization, and monitoring of the National Access to Justice Strategy. Given that the results will be of interest to those involved in impact evaluations globally (not only those involved in MCLE and Indonesia) PREM Poverty and the Access to Justice teams will disseminate the results to a wide audience of evaluation and judicial reform practitioners. 3. Results from the impact evaluation will be publicized at conferences with donors to build support for legal reform projects. In particular, this work will inform a joint AUSAID/World Bank effort to improve the analytical underpinnings of Justice for the Poor projects in the region. 4. Results from the impact evaluation should be published in academic journals. Less technical explanations should be published as a World Bank paper/report for internal and external audiences. Workplan The impact evaluation will be conducted in eight phases: Phase 1: Instrument Development / Recruitment / Training

Develop field guide and research instruments (we are currently completing this phase) Recruit qualitative field researchers Provide training to field researchers

Phase 2A: Baseline Quantitative Data Collection

Household survey Key informant survey Community survey

16

Phase 2B: Baseline Fieldwork

Field trip #1: Village mapping Debriefing, preliminary analysis and presentations Field trip #2: Case study collection

Phase 3: Baseline Write-up

Quantitative data entry and cleaning Field reports Case study reports

Phase 4: Analysis

Integration of baseline quantitative and qualitative data Overall report Presentation of findings

Phase 5: Community Ethnographies

Initial fieldwork In-depth fieldwork #1 In-depth fieldwork #2 Write-up

Phase 6A: Endline Quantitative Data Collection

Household survey Key informant survey Community survey

Phase 6B: Endline Fieldwork

Preparations (i.e. re-recruitment and training as necessary) Review field guide and research instruments Field trip #1: Village mapping Debriefing, preliminary analysis and presentations Field trip #2: Case study collection

Phase 7: Endline Write-up

Quantitative data entry and cleaning Field reports Case study reports

Phase 8: Analysis

Integration with quantitative data and report writing Final reports Presentations

Timeline

17

The impact evaluation will be implemented over a period of approximately two to fours years (late 2007 to mid 2010 or 2012) depending on funding and how long the program runs. This timeline only covers the first 2½ years from late 2007 to 2010. As the table below indicates there are two distinct periods of intensive work. The initial preparations, baseline fieldwork and analysis will be done between December 2007 and August 2008. This schedule will be linked to the quantitative research. Both the qualitative and quantitative research are dependent on the program implementation timeline.

Year 2007 2008 2010 Month Dec Jan – Aug Sept Oct Nov Dec Jan Feb Mar Apr Phase 1 (Preparations) X X Phase 2A (Data Collection) X X X X X Phase 2B (Baseline fieldwork)

X X X X X

Phase 3 (Baseline write-up) X X X Phase 4 (Baseline analysis) X X Phase 5 (Ethnography) X Phase 6A (Data Collection) X Phase 6B (Endline fieldwork) X Phase 7 (Endline write-up) X Phase 8 (Overall analysis) X Impact Evaluation Team Government team: Verdi Yusuf (KPDT) Umi (Bappenas) M&E Specialist (NMC-SPADA) Others (tbc) WB Jakarta team: Matthew Zurstrassen Ikhsan Nurul Samuel Clark John Voss Susan Wong WB Washington DC team: Kathleen Beegle Penelope Brown Markus Goldstein Kristen Himelein Caroline Sage Emmanuel Skoufias

18

19

Annex 1 Treatment Assignment and Sample Selection MCLE Evaluation – Aceh and Maluku, Indonesia Original Draft : April 1, 2008, revised: August 1, 2008, revised: September 2, 2008 The program treatment assignment and sampling design for the SPADA-MCLE evaluation in Aceh and Maluku was done in March 2008 by a team consisting of Markus Goldstein (DEC / AFTPM), Emmanuel Skoufias (PREM), Samia Amin (PREM), Samuel Clark (EASSO), Kristen Himelein (DECRG) and Steven Shewfelt (Yale University), and based on advice from Juan Munoz (Sistemas Integrales). The Aceh sample was revised in July and August 2008 based on changes in funding availability. The Maluku sample was revised in August 2008 due to the objection from the survey firm to those locations in Maluku Tenggara that do not have public transportation. The sample, this report and the subsequent revisions were prepared by Kristen Himelein.

Program Design

Selection of the treated villages was done based on program guidelines agreed upon by the World Bank and the government of Indonesia. The project design indicates that a specified number of sub-districts be chosen in each of four government-selected districts in Aceh and two in Maluku to receive community legal post (posko) to work at the sub-district level. Within the chosen sub-district, a specified number of villages will also receive two paralegals to work at the village-level. Initially four sub-districts will be chosen for each participating district, with one additional sub-district in Aceh Barat and three additional sub-districts in each Aceh Besar, Aceh Utara, and Pidie / Pidie Jaya to follow if additional funds become available.

Province District # Kecamatan

(Posko) # Villages # Paralegals

Aceh Barat 4 32 64 Aceh Besar 4 32 64 Aceh Utara 4 32 64 Pidie

Aceh

Pidie Jaya* 4 32 64

Maluku Tengah Seram Bagian Barat* Seram Bagian Timur*

4 32 64

Maluku Tenggara

Maluku

Kepulauan Aru* 4 32 64

Total: 2 6 24 192 384 * These administrative have recently split from the main districts. For the purposes of this evaluation, they will be treated as part of the original district.

Sample Size As previously detailed in a design note from Markus Goldstein, calculations indicated a sample size of 400-450 households per treatment arm (with two treatment arms in Aceh and one treatment arm in Maluku) would be appropriate given the assumptions on expected outcome and intra-household correlation. Due to cost considerations and in consultation with

20

the survey firm, it was decided to opt for 10 households per cluster in Maluku (45 total clusters) and 6 households per cluster in Aceh (66 total clusters). Treatment Assignment In Aceh, four sub-districts were randomly assigned to the treatment in Aceh Barat, Aceh Besar, Aceh Utara and Pidie. In Maluku, four districts were randomly assigned to the treatment in both Maluku Tengah and Maluku Tenggara. The table below summaries the probability of an individual sub-district being chosen in a given district.

Province District Sub-District Selected Sub-District Total Percentage Aceh Barat 4 9 44.44% Aceh Besar 4 22 18.18% Aceh Utara 4 21 19.05%

Aceh

Pidie 4 30 13.33% Maluku Tengah 4 19 21.05% Maluku Maluku Tenggara 4 13 30.77%

Total: 2 6 24 114

Due to the previously mentioned funding limitations, the current treatment in Aceh has been scaled back from original plans. Therefore one sub-district in Aceh Barat, and three sub-districts each in Aceh Besar, Aceh Utara and Pidie are held in reserve for future funding. They will not be eligible to be part of the sample for either the treatment or control groups.

Following the selection of the sub-district, eight villages were randomly selected within each selected sub-district to receive the paralegal treatment. Therefore, 32 villages should be assigned to receive the paralegal treatment in each district, though some sub-districts do not have a sufficient total number of villages.

Province District Sub-District Villages Selected Villages to be Selected Village Total Percentage Arongan Lambalek 8 19 42.11% Kaway Xvi 8 63 12.70% Sama Tiga 8 27 29.63% Woyla 8 43 18.60%

Aceh Barat

Sungai Mas* 8 14 57.14% Indrapuri 8 50 16.00% Lembah Seulawah 8 12 66.67% Leupung 5 5 100.00% Lhoong 8 25 32.00% Darussalam* 8 22 36.36% Kota Jantho* 8 13 61.54%

Aceh Besar

Baitussalam* 8 13 61.54% Kuta Makmur 8 49 16.33% Nibong 8 19 42.11% Samudera 8 39 20.51% Tanah Luas 8 55 14.55% Baktiya Barat 8 26 30.77% Langkahan 8 22 36.36%

Aceh Utara

Seunudon 8 28 28.57% Geumpang 5 5 100.00% Glumpang Baro 8 18 44.44% Grong Grong 7 7 100.00% Pidie 8 61 13.11% Tangse* 8 28 28.57%

Aceh

Pidie

Batee* 8 28 28.57%

21

Mane* 4 4 100.00% Total: 121 76

* Villages held in reserve for additional funding.

Banda 10 8 80.00% Pulau Gorom 20 8 40.00% Saparua 16 8 50.00%

Maluku Tengah

Seram Timur 17 8 47.06% Aru Tengah 45 8 17.78% Pulau Dullah Selatan 5 5 100.00% PP Kur 11 8 72.73%

Maluku

Maluku Tenggara

Tayando Tam 6 6 100.00% Total: 59 In three sub-districts in Aceh, Leupung, Geumpang and Grong Grong, and two sub-districts in Maluku, Tayando Tam and Pulau Dullah Selatan, there are fewer villages than the eight necessary for the program. In these cases, all available villages will be treated. As to the remaining seven villages in Aceh needed for a complete assignment of the 128 total villages, and the five needed for Maluku for a total of 64, they will be chosen in such a way to minimize implementation costs without contaminating the control villages. As they are not randomly selected, they will not be included in the evaluation. Excluded Villages in Maluku A number of villages were excluded from the potential pool of treatment and control sample villages by survey firm as they are not accessible by commercial transportation. Excluded from the sample by the survey firm are the following sub-districts: all of Aru Selatan, all of Aru Tengah (except Benjina, Gardakau, Maririmar, Namara, Papakula, Selibata-Bata and Selilau), all of Pulau-Pulau Aru (except Galai Dubu, Gorar, Jabulenga, Karangguli, Siwalima, Tungu, Tunguwatu, Ujir and Wangel), and all of PP Kur. These villages were not excluded from the assignment of the program because it is government funded and villages cannot be excluded from eligibility based on location. MCLE Treatment Sample Selection The samples for each arm of the evaluation were selected using probability proportional to size [pps] sampling. The samples were not stratified by district or sub-district. In Aceh, out of the universe of 121 possible paralegal treatment villages, pps sampling was used to select 66 clusters for the sample. As a village may be selected more than once under pps sampling, a total of 66 clusters in 56 villages were selected. Additionally, in Aceh, a sample was selected from the posko-only treatment group. This sample was selected using pps sampling from all villages within the sub-districts receiving the program that were not selected into the paralegal treatment group. In case, 66 clusters in 66 unique villages were selected. Following the exclusions in Maluku, there are 32 possible treatment villages in Maluku Tengah and 11 in Maluku Tenggara. Sixteen treatment villages in Maluku Tenggara are excluded. Out of the universe of 43 possible treatment villages, pps sampling was used to

22

select 45 clusters in 30 unique villages for the sample. A separate posko-only sample was not selected as there are not a sufficient number of villages. Households within each treatment village will be selected using simple random sampling from available village lists. A complete list of villages and populations is attached as an appendix to this note. CRT Treatment Selection In addition to the MCLE program in Aceh, an additional Conflict Resolution Training program will also be evaluated using the same control sample as the MCLE evaluation. For the CRT sample, two sub-districts were selected in Aceh Barat, and four from each Aceh Besar, Aceh Utara and Pidie. The sub-districts selected for the CRT sample are Panta Ceureumen and Woyla Timur in Aceh Barat, Ingin Jaya, Kuta Cot Glie, Pulo Aceh and Seulimeum in Aceh Besar, Dewantara, Matangkuli, Muara Batu, Tanah Jambo Aye in Aceh Utara, and Bandar Dua, Kota Sigli, Simpang Tiga and Tiro/Truseb in Pidie. Ten villages were randomly selected within the selected sub-districts to receive the CRT training program. Of the possible universe of 140 CRT treatment villages, 62 were chosen using pps sampling for the CRT treatment sample. Control Sample Selection Different methods were used to select the control samples in Aceh and Maluku due to the presence of the CRT program in Aceh and infrastructure considerations in Maluku. The original preferred control sample method was to use pps sampling to select villages from the pool of all villages which were not selected for the posko or paralegal treatments. Due to high transportation costs and geographic separation, particularly in Maluku, this design was judged to be unfeasible due to cost. In Aceh, the alternative employed was to select the control sub-districts from those not assigned to the MCLE or CRT treatment groups. This left two from Aceh Barat and four from each Aceh Besar, Aceh Utara and Pidie. The eligible control sub-districts are Bubon and Woyla Barat in Aceh Barat, Darul Imarah, Darul Kamal, Kruent Barona Jaya, and Suka Makmur in Aceh Besar, Baktiya, Cot Girek, Meurah Mulia, and Paya Bakong in Aceh Utara, and Bandar Baru, Indrajaya, Peukan Baro, and Sakti in Pidie. The villages from these sub-districts form the total pool of villages from which pps sampling was used to select control clusters. The 66 selected clusters were in 62 unique villages. In Maluku, two main concerns prevented the use of the use of the same method: (1) cost due to limited infrastructure, and (2) a lack of a sufficient number of villages. Instead, pps sampling was performed at two levels: first at the sub-district then at the village level. This was done to construct a sample that matches the overall population as closely as possible while remaining within the boundaries of a reasonable budget and avoiding those villages excluded by the survey firm. Therefore a pps sample of eight sub-districts was drawn based on the total population of the villages in both Maluku Tengah and Maluku Tenggara in the sub-districts that were not assigned to receive the MCLE treatment. The sub-districts selected were six in Maluku Tengah (Amahai, Huamual Belakang, Leihitu, Sala Hutu, Seram Barat and Werinama), and two in Maluku Tenggara (Pulau Dullah Utara and Pulau Pulau Aru).

23

Then, individual villages were sampled from within the above pool selected in the first stage for sampling. From a possible 90 villages (73 in Maluku Tengah and 13 in Maluku Tenggara), 45 clusters were selected in 38 unique villages. By first selecting sub-districts from which the control sample is drawn, costs are decreased by centralizing the locations to which the survey teams will have to travel. This is done, however, at a trade-off of increased sampling error. Due to the nature of the restrictions placed on the assignment of the treatment by the program design, the treatment and control groups are significantly different in their structure. Because, for example, a far higher percentage of total villages will receive paralegals in the district of Aceh Barat (13% compared to about 5% in the other treatment districts), Aceh Barat will be over-represented in the treatment sample as compared to the population as a whole. The structure of the control weights will also differ between Aceh and Maluku. Because the selection probabilities are known at each level of both the treatment and sample selection process, it will be possible to correct for these differences ex-post using weights. Additionally, the villages that were not selected for the first round of treatment, but also not included in the potential pool for the control sample must be taken into account when making these calculations. An Excel file for each district including a population table is listed as an appendix to this note from which the weights can be constructed. Note on Conflict Resolution Training During the planning stages, it was also discussed to implement the CRT training in a sub-set of villages in the MCLE treatment group to study the interaction between the two programs. Due to concerns, however, about limited power to detect additional effects in the sub-sample, and concerns about “noise” in the main evaluation sample, the team decided to keep the treatment groups for the two evaluations completely separate.

24

Annex2

NO SUB TOTAL (Rp.) SUB TOTAL ($)

Per Budget Item1 Quantative Baseline 191,635

1.1 Preparation and Recruitment 102,798,000 11,1741.2 Training 387,724,000 42,1441.3 Survey Implementation 1,240,208,400 134,8051.4 Data Cleaning and Consolidation 32,308,000 3,512

2 Quantative Endline 255,0662.1 Preparation and Recruitment 102,798,000 14,8722.2 Training 387,724,000 56,0942.3 Survey Implementation 1,240,208,400 179,4262.4 Data Cleaning and Consolidation 32,308,000 4,674

3 Qualitative Baseline 105,9843.1 Consultant Fees 586,500,000 63,7503.2 Recruitment Costs 26,400,000 2,8703.3 Training 93,150,000 10,1253.4 Fieldwork 180,000,000 19,5653.5 Fieldwork Supervision 16,200,000 1,7613.6 Write-up Workshop 72,800,000 7,913

4 Qualitative Endline 151,6254.1 Consultant Fees 659,500,000 95,4124.2 Recruitment Costs 26,400,000 3,8194.3 Training 93,150,000 13,4764.4 Fieldwork 180,000,000 26,0414.5 Fieldwork Supervision 16,200,000 2,3444.6 Write-up Workshop 72,800,000 10,532

5 Staff Inputs 254,3445.1 Staff Costs 221,6685.2 Supervision and Travel 32,676

8,819,612,660 958,654 * Price index 10% p/a over 3 years

ITEM

MCLE Impact Evaluation - Budget2008-2011

Total Budget 2008-11

Budget Notes: 5 Staff Inputs. * Sources of funding. Washington DC Staff from PRMPR and DEC funded by Bank Budget excluding Research Assistant who is funded by GNPP. * Budget for 2008 – 2009. Note rest of budget runs 2008 – 2011.