MID-TERM EVALUATION REPORT

DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

MID-TERM EVALUATION REPORT

ADVANCING COMMUNITY EMPOWERMENT IN SOUTHEASTERN MYANMAR (ACE) February 24, 2021

KELLY RAMUNDO FOR USAID

ACE Mid-Term Evaluation Abstract

Social Impact conducted a Mid-Term Evaluation of the Advancing Community Empowerment in Southeastern Myanmar (ACE) to review progress in its first three years. ACE is a five-year $48 million project USAID in Burma implemented by a consortium led by Pact, including Mercy Corps, Save the Children International and Community Partners International. Six evaluation questions examined the impact of political context on the project, progress against expected outcomes, the project’s approach, its operational structure, the effect of the COVID-19 pandemic, and recommendations on possible changes for its remaining two years. The team conducted the evaluation remotely due to the pandemic and used a mixed methodology of key informant interviews and a mini-survey of village-level direct beneficiaries. While the evaluation team was asked to examine ACE’s multi-sectoral and integrated approach, they found it was a partner-specified rather than a project-wide approach. The evaluation team found that ACE had mixed results in reducing community vulnerabilities, strengthening community participation, and strengthening local governance systems, with different components having met, exceeded, or underperformed on their targets. ACE experienced slow implementation due to the pandemic, which significantly affected the ability to use traditional community organizing techniques, as well as significant turnover of senior management staff in the early part of the project. With the pandemic expected to continue into 2022, ACE will require significant revisions to its expected outcomes, timeline, and activities.

USAID.GOV USAID BURMA ACE EVALUATION REPORT | i

CONTENTS

CONTENTS ........................................................................................................................................................................ I

FIGURES .............................................................................................................................................................................. II

TABLES................................................................................................................................................................................. II

ACRONYMS ...................................................................................................................................................................... III

EXECUTIVE SUMMARY ..................................................................................................................................................V

Evaluation Purpose ........................................................................................................................................................ v

Evaluation Questions .................................................................................................................................................... v

Project Background ..................................................................................................................................................... vi

Evaluation Methods and Limitations ........................................................................................................................ vi

EVALUATION PURPOSE & EVALUATION QUESTIONS .................................................................................... 1

Evaluation Purpose ........................................................................................................................................................ 1

Evaluation Questions .................................................................................................................................................... 1

PROJECT BACKGROUND ............................................................................................................................................ 3

Background ..................................................................................................................................................................... 3

Project Goal and Objectives....................................................................................................................................... 4

Theory of Change and Critical Assumptions.......................................................................................................... 5

EVALUATION METHODS & LIMITATIONS ............................................................................................................ 6

Context ........................................................................................................................................................................... 6

Document Review......................................................................................................................................................... 6

Key Informant Interviews ............................................................................................................................................ 7

Mini-Survey ..................................................................................................................................................................... 8

FINDINGS, CONCLUSIONS, & RECOMMENDATIONS ................................................................................... 11

Overview ....................................................................................................................................................................... 11

Evaluation Question 1 — Externalities .................................................................................................................. 11

Evaluation Question 2 — Project Achievements ................................................................................................ 15

Evaluation Question 3 — Project Approach ........................................................................................................ 24

Evaluation Question 4 — Project Structure ......................................................................................................... 28

Evaluation Question 5 — Pandemic Impact ......................................................................................................... 31

Evaluation Question 6 — Recommendations ...................................................................................................... 34

ANNEXES ......................................................................................................................................................................... 39

Annex I: Evaluation Statement of Work ................................................................................................................ 39

Annex II: Sources of Information ............................................................................................................................ 46

Annex III: Data Collection Instruments ................................................................................................................. 50

Annex IV: Disclosure of Any Conflicts of Interest .............................................................................................. 79

ii | USAID BURMA ACE EVALUATION REPORT USAID.GOV

FIGURES

Figure 1. ACE Logic Model Y1–Y3 ................................................................................................................................ 5

Figure 2. Participation by Sex in GCE Groups ......................................................................................................... 17

Figure 3. Women’s Participation in Health & WASH Committees..................................................................... 18

Figure 4. Level of Women's Participation in PTAs .................................................................................................. 19

Figure 5. Impact of COVID on WORTH Group Meetings ................................................................................... 21

Figure 6. Impact of COVID on Health & WASH Group Meetings ..................................................................... 22

Figure 7. Participation of Women in Health and WASH Groups........................................................................ 23

Figure 8. Frequency of Village Meetings Before and After COVID ..................................................................... 34

TABLES

Table 1. Typology Stakeholders Targeted for Key Informant Interviews ............................................................ 7

Table 2. Mini-Survey Sample Summary ......................................................................................................................... 9

USAID.GOV USAID BURMA ACE EVALUATION WORKPLAN | iii

ACRONYMS

ACE Advancing Community Empowerment

ACESM Advancing Community Empowerment in Southeastern Myanmar

AOR Agreement Officer’s Representative

APEA Applied Political Economy Analysis

CBO Community-Based Organization

CNA Community Needs Assessment

COP Chief of Party

COR Contracting Officer’s Representative

COVID Coronavirus Disease

CPI Community Partners International

CSG Critical Service Grant

CSO Civil Society Organization

DCOP Deputy Chief of Party

DGHA Democracy, Governance, and Humanitarian Assistance

DRR Disaster Risk Reduction

EAO Ethnic Armed Organization

EDW Evaluation Design and Workplan

EHO Ethnic Health Organization

EICS Equity, Inclusion, and Conflict Sensitivity

EQ Evaluation Question

ET Evaluation Team

FGD Focus Group Discussion

GBV Gender-Based Violence

IBN Interest-Based Negotiation

IDP Internally Displaced Persons

IKG In-Kind Grant

INGO International Non-Governmental Organization

IR Intermediate Result

IRB Institutional Review Board

ITHP Inclusive Township Health Plan

iv | USAID BURMA ACE EVALUATION REPORT USAID.GOV

KII Key Informant Interview

LIFT Livelihoods and Food Security Fund

LOP Life of Project

MEL Monitoring, Evaluation, and Learning

MHF Myanmar Humanitarian Fund

NCA Nationwide Ceasefire Agreement

NLD National League for Democracy

NOFO Notice of Funding Opportunity

OCHA Office for the Coordination of Humanitarian Affairs

PPE Personal Protective Equipment

PTA Parent Teacher Association

RFA Request for Applications

RHELS Responsive Health, Education, and Livelihoods Services

SCI Save the Children International

USAID United States Agency for International Development

USDP Union Solidarity and Development Party

VCG Village Coordination Groups

VDC Village Development Committee

VDF Village Development Fund

VHC Village Health Committee

VWC Village WASH Committee

WASH Water, Sanitation, and Health

WORTH Pact’s women’s economic empowerment program

Y1 Year One

Y2 Year Two

Y3 Year Three

USAID.GOV USAID BURMA ACE EVALUATION REPORT | v

EXECUTIVE SUMMARY

EVALUATION PURPOSE

The Advancing Community Empowerment (ACE) in Southeastern Myanmar is a five-year $48 million project of the United States Agency for International Development (USAID) in Burma. It is implemented by a consortium led by Pact that includes Mercy Corps (MC), Save the Children International (SCI), and Community Partners International (CPI). This mid-term evaluation covers the period from September 2017 until September 2020. The purpose of the evaluation is to assess ACE's progress towards achieving its intended outcomes, intermediate results, objectives, and overall goals in the current political context and after the impact of the COVID-19 pandemic. USAID/Burma asked the evaluation team (ET) to provide recommendations for the remaining two years of the ACE project and future programming.

EVALUATION QUESTIONS

This evaluation's primary audience is USAID/Burma, specifically, the Office of Democracy, Governance, and Humanitarian Assistance (DGHA), Office of Public Health, and Program Development Office, as well as the ACE implementors. USAID/Burma outlined the following six Evaluation Questions (EQs) to guide the ET:

I. How have externalities related to the evolving political context in ACE implementation areas impacted ACE's ability to achieve expected objectives over the first three years of the project?

II. To what extent have ACE's health, education, livelihoods, water, sanitation, and hygiene (WASH), and disaster risk reduction (DRR) interventions contributed to achieving its expected outcomes, intermediate results, and objectives at this point in the project implementation? Specifically, in the following areas:

a. Health care access and coordinated service delivery improved b. Education access and services improved c. Sustainable livelihoods promoted d. Water, sanitation, and hygiene improved e. Community capacity for preparedness, mitigation, and response to disasters increased

III. To what extent has a multi-sectoral and integrated community development approach contributed to achieving expected objectives, intermediate results, and outcomes in the areas of:

a. Reducing Community Vulnerabilities; b. Strengthening Community Participation; and, c. Strengthening Local Governance Systems?

IV. How has the operational structure of the ACE project (a combination of direct implementation consortium partners and sub-partners grants) affected ACE's ability to achieve desired outcomes and results at this point in the project implementation?

a. From the perspective of the prime implementing partner b. From the perspective of the consortium partners c. From the perspective of the local partners d. From the perspective of the direct beneficiaries

V. How has the COVID-19 pandemic:

a. affected ACE's ability to meet expected outcomes, intermediate results, and objectives?

vi | USAID BURMA ACE EVALUATION REPORT USAID.GOV

b. resulted in the modifications to respond to the needs which arose out of the pandemic?

VI. Where are the potential opportunities to alter, scale-up, reinforce, or cancel activities or approaches over the remainder of the program in response to the changing context and political dynamics in the implementation areas.

PROJECT BACKGROUND

As it took power in 2016, the National League for Democracy (NLD) government inherited a complex set of challenges. Communities in conflict-affected areas, including in southeastern Myanmar, still faced "obstacles to fully benefiting from the transition due to their remoteness, level of poverty, lack of basic infrastructure, presence of ongoing conflict, pervasive mistrust, contested governance systems, or other barriers."1 The challenge of negotiating new power-sharing and governance arrangements as part of the peace process remained and had not yet produced tangible development results. Meanwhile, an estimated 400,000 internally displaced persons (IDPs) remained in southeastern Myanmar, and more than 100,000 refugees remained in Thailand. At the start of the ACE project, USAID saw this as an opportune time "for assistance, engagement, and policy support to influence the trajectory of reforms."

USAID designed the ACE project to build upon its previous assistance to communities along the Myanmar-Thailand border, using a conflict- and context-sensitive approach assuming "continued momentum and progress on the peace process and democratic reforms." Using sub-grants as a critical tool, ACE delivers essential services "while modeling democratic principles of local governance and increasing the capacity of communities, civil society, and local decision-makers in conflict-affected areas to constructively engage with each other."

ACE's goal is that vulnerable communities will be empowered to engage in local decision-making processes. To achieve this, the project's objectives and intermediate results (IR) are:

Objective I: Community Vulnerabilities Reduced

Objective II: Community Participation Strengthened

Objective III: Local Governance Systems Strengthened

EVALUATION METHODS AND LIMITATIONS

The ET conducted this evaluation during the 2020 COVID-19 pandemic when strict government restrictions on travel were in place. The team implemented it entirely remotely, with data collected through a document review, 43 key informant interviews, and a mini-survey of 186 village level volunteers working with the ACE project conducted over the phone.

This methodology had certain limitations. Tools typically used in such evaluations, such as focus group discussions and direct observation, were not available to the ET to develop a deeper understanding of the results of the project. Only respondents with reliable phone access could be included in the mini-survey, even though ACE supports some remote communities. Not capturing the views of the most remote or those who did not answer their phone or agree to participate, could heighten the response bias in this evaluation. There is an inherent bias as only respondents who were available and willing to respond were included in the survey. The views of those who were unwilling, unavailable, or unreachable because of their phone access were left out. The ET recognized that the remote methodology increases this bias

1 Notice of Funding Opportunity (NOFO) Number: RFA-482-17-000004, USAID Burma, March 14, 2017.

USAID.GOV USAID BURMA ACE EVALUATION REPORT | vii

because it is easier to refuse, miss, or never receive a request to be interviewed. Nonetheless, the evaluation provides useful information to guide future decision-making.

EVALUATION QUESTION 1- EXTERNALITIES

Political externalities affect the ACE project from the national, regional, and local levels. The NLD came to power in 2015 and promised to make peace with ethnic groups a national priority. However, during most of ACE’s project period, the peace process has been deadlocked. The lack of progress meant that conflict continued in the Southeast, causing localized instability. There was no consistent guidance from the central government on implementing the National Ceasefire Agreement. The commitments in this agreement did not materialize into tangible work with EAOs in health, education, and socio-economic development. There were no national level policies and practices handed to local administrators that could be converted into concrete change on the ground.

Without the steadying hand from such high-level guidance, decision making often defaulted to state governments and regional military commanders. This resulted in new local conflicts and clashes that further eroded trust between the government and ethnic groups. As the national government failed to prioritize peace and the convergence of the two political systems, the ACE project faced challenges working toward project objectives. Decisions on government coordination and cooperation with ethnic service providers were often left to the discretion of individual township-level.

The lack of inertia in the peace process left ACE without a favorable environment to implement the project. Localized conflict continued in the Southeast during the project’s first three years, leading to greater mistrust, and new restrictions on NGOs made bringing the two sides together more difficult and slowed down ACE’s implementation. At the local level, the absence of clear national priorities and policies left decisions on cooperation between government and ethnic groups to local administrators. This left ACE dependent on personal relationships with officials rather than government policy. Each time the central government rotated local personnel, the project lost time building relationships with the new officials. Collectively, these political externalities undermined ACE’s ability to achieve its objectives of reducing community vulnerabilities and strengthening participation, as well as undermined its objective to strengthen local government systems.

EVALUATION QUESTION 2 – PROJECT ACHIEVEMENTS

ACE has made varied progress towards its Objective I by reducing community vulnerabilities. ACE is on target for some indicators, has over-achieved on some, and is behind on others. The project has made many discrete contributions in its various areas of service delivery. ACE is responsible for training health care workers and improving access to services for women and children. It has trained teachers and improved the quality of literacy materials provided to students. ACE has contributed to better livelihoods by training farmers, improving value chains, and giving village women access to small scale finance. In the WASH sector, ACE has provided access to better sanitation facilities to thousands by building the capacity of hundreds of village-level volunteers.

ACE has made progress towards its Objective II of strengthening community participation by organizing new structures at the village level, training its leaders, developing community action plans, and working to ensure women and youth can participate in decision-making. It has faced more significant challenges in achieving Objective III of strengthening local government systems because the political context has not been favorable to the government and ethnic service providers working together.

The project’s achievement of its objectives was slowed down by the region's political complexity, even before the pandemic reached Myanmar. The pandemic restrictions on travel and meetings further delayed activities across the board. The restrictions’ impact most significantly affected the project’s disaster risk

viii | USAID BURMA ACE EVALUATION REPORT USAID.GOV

reduction work, stopping it just as it was starting. The evaluation found evidence that the project’s work with village-level institutions helped decision-making become more democratic and inclusive, particularly through its high level of women’s involvement. Comparatively, the project has struggled to bring government and ethnic service providers together.

EVALUATION QUESTION 3 – PROJECT APPROACH

The ACE consortium takes a multi-sector approach through its integrated community development at the village level, as implemented by Pact. The concept of multi-sectoral, integrated community development is understood differently by members of the consortium. At the consortium level, members see their work together as coordinated but not integrated; each organization, for the most part, implements work in its sector independently. Pact is implementing two multi-sectoral models in its village-level work, one based around integrated services and the other on community leadership. The ET was unable to determine the relative effectiveness of the two approaches with the available data.

The lack of a shared understanding of what a multi-sector approach means for the consortium members and their individual activities' dispersed nature has contributed to a weak sense of common purpose among partners.

EVALUATION QUESTION 4 – PROJECT STRUCTURE

From the prime partner's perspective, the project structure is a good one. However, some ACE stakeholders question, particularly after the pandemic, whether ACE has struck the right balance between direct implementation and working through local partners. Day-to-day, the partners find the consortium structure complicated and often bureaucratic. Partners perceived the high levels of senior staff turnover during the early years of the project to be significantly disruptive. Staff saw co-location in the field offices as helpful. Still, co-location only went so far in overcoming the consortium's siloed structure. Local partners welcomed ACE's resources and expertise but found the grant dispersal process slow, and USAID restrictions on activities too limiting.

According to respondents, the pandemic, local-level conflict, and challenges with management turnover were the most obvious factors limiting the ACE project’s progress. It is not clear how the project’s structure affected results, given the overwhelming influence of these factors.

EVALUATION QUESTION 5 – PANDEMIC IMPACT

The COVID-19 pandemic has dramatically impacted project operations. Many activities based around community organizing were suspended from Q2 Y3. The project has adapted to working remotely, where it can, and started to alter some of its activities to shift resources to help partners with their pandemic response.

Despite positive momentum, the ACE consortium is unlikely to achieve all of its intended results within the allocated timeframe due to the pandemic. With growing consensus in the development community that such restrictions will continue for some time, ACE may soon need to make significant changes to its design and activities.

EVALUATION QUESTION 6 – RECOMMENDATIONS

While the project has started to adapt to the new environment by shifting resources to pandemic response, it has not yet formally changed its outcomes, objectives, intermediate results, or core activities. USAID and USG restrictions limit ACE’s programming options, while the pandemic restrictions limit ACE’s ability to scale up through increased direct implementation. The project could continue to look for

USAID.GOV USAID BURMA ACE EVALUATION REPORT | ix

opportunities to increase its resource allocation to local partners, even though many face the same restrictions as endured by ACE. It could expand ongoing plans to reinforce existing sub-grant mechanisms by adding additional opportunities available for local partners. With no foreseeable end to pandemic restrictions, ACE may be running out of time.

Expected results from the project will need to be revised, and some project funding reallocated if ACE is to make the best use of its remaining time and budget. Some pre-existing sub-grant mechanisms, especially those with partners who have shown resilience during the pandemic, could be scaled up. ACE could allocate more funding for pandemic response to existing sub-grantees. USAID/Burma and ACE should conduct scenario planning to assess which activities will be possible if COVID restrictions continue throughout the performance period. If pandemic restrictions continue, ACE may need to cancel specific project components as they may no longer be viable.

USAID.GOV USAID BURMA ACE EVALUATION REPORT | 1

EVALUATION PURPOSE & EVALUATION QUESTIONS

EVALUATION PURPOSE

This document describes the results of Social Impact’s (SI) midterm performance evaluation of the Advancing Community Empowerment (ACE) in Southeastern Myanmar project, which is implemented by a consortium of partners led by Pact, Inc. The United States Agency for International Development (USAID) in Burma made the $48 million, five-year award in August 2017. ACE implements activities in Kayah, Kayin, Mon, and southern Shan states as well as supporting sub-grants in other locations.

The purpose of the evaluation is to assess the progress of ACE towards achieving its intended outcomes, intermediate results, objectives, and overall goals relative to the current political context and impacts of COVID-19; to identify best practices and lessons learned related to health, education, and multi-sectoral/integrated approaches to improving governance; and to provide recommendations for the remaining two years of the ACE project and future programming.

The primary audience for this evaluation is USAID/Burma, specifically the Office of Democracy, Governance, and Humanitarian Assistance (DGHA), Office of Public Health, and Program Development Office. Relevant components and information from the evaluation will be shared with implementing partners, sub-partners, and the broader development community at the Mission’s discretion.

EVALUATION QUESTIONS

The original scope of work for this evaluation (see Annex I: Evaluation Statement of Work) contained six Evaluation Questions (EQs) for the Evaluation Team (ET) to answer. After initial consultations with USAID/Burma and the ACE team some revisions were proposed and explained in the Evaluation Design and Workplan. The approved EQs are:

1. How have externalities related to the evolving political context in ACE implementation areas impacted ACE’s ability to achieve expected objectives over the first three years of the project?

2. To what extent have ACE’s health, education, livelihoods, water, sanitation, and hygiene (WASH), and disaster risk reduction (DRR) interventions contributed to achieving its expected outcomes, intermediate results, and objectives at this point in the project implementation? Specifically, in the following areas:

a. Health care access and coordinated service delivery improved b. Education access and services improved c. Sustainable livelihoods promoted d. Water, sanitation, and hygiene improved e. Community capacity for preparedness, mitigation, and response to disasters increased

3. To what extent has a multi-sectoral and integrated community development approach contributed to achieving expected objectives, intermediate results, and outcomes in the areas of:

a. Reducing Community Vulnerabilities; b. Strengthening Community Participation, and; c. Strengthening Local Governance Systems?

4. How has the operational structure of the ACE project (a combination of direct implementation consortium partners, and sub-partner grants) affected ACE’s ability to achieve desired outcomes and results at this point in the project implementation?

2 | USAID BURMA ACE EVALUATION REPORT USAID.GOV

a. From the perspective of the prime implementing partner b. From the perspective of the consortium partners c. From the perspective of the local partners d. From the perspective of the direct beneficiaries

5. How has the COVID-19 pandemic:

a. affected ACE’s ability to meet expected outcomes, intermediate results, and objectives? b. resulted in the modifications to respond to the needs which arose out of the pandemic?

6. Where are the potential opportunities to alter, scale-up, reinforce, or cancel activities or approaches over the remainder of the program in response to the changing context and political dynamics in the implementation areas?


PROJECT BACKGROUND

BACKGROUND

The ACE project originated during a time of transition for Myanmar. With the National Ceasefire Agreement (NCA) signed in 2015 by the military led government, USAID and other donors saw an opportunity to assist the country’s transition from decades of conflict. When the National League for Democracy (NLD) took office in April 2016, President Obama hailed this a “historic democratic transfer of power to a civilian-led government.” The United States was committed to “help the people and Government of Burma achieve a more inclusive, peaceful, and prosperous future,” he said.2 In March 2017, USAID/Burma requested proposals for the Advancing Community Empowerment in Southeastern Myanmar program, known by the acronym ACE.3 The democratically elected NLD government had then been in office for less than one year and for the international community this was a time of great optimism.

In 2017, the theory of change underpinning USAID’s strategy to strengthen democratic governance during the transition assumed “the elected government maintains constructive engagement with all interested parties with a stake in the reforms,” including in the peace process started by the previous military-led administration.4 The ACE project's goal was “to empower vulnerable communities by mitigating their vulnerabilities, encouraging robust community participation in decision making, and strengthening mechanisms for more accountable local governance.”5

As it took power in 2016, the NLD government inherited a complex set of challenges. Communities in conflict-affected areas, including in the southeast, still faced “obstacles to fully benefiting from the transition due to their remoteness, level of poverty, lack of basic infrastructure, presence of ongoing conflict, pervasive mistrust, contested governance systems, or other barriers.”6 USAID had long worked from Thailand to support the humanitarian needs of refugees, migrants, and internally displaced persons (IDPs) along the Thailand-Myanmar border, and these needs remained even after the new government was in place. The challenge of negotiating new power-sharing and governance arrangements as part of the peace process remained and had not yet produced tangible development results. Meanwhile, an estimated 400,000 IDPs remained in southeastern Myanmar and more than 100,000 refugees remained in Thailand. At the start of the ACE project, USAID saw an opportune time “for assistance, engagement, and policy support to influence the trajectory of reforms.”7

Over decades of conflict and military rule, in many areas controlled by Ethnic Armed Organizations (EAOs), public services such as primary healthcare and basic education have been provided by non-state actors. Ethnic service providers, such as ethnic health organizations (EHOs) and community-based organizations (CBOs), are the primary service providers for local communities and displaced persons, often with short-term support from private or international donors. Where these EAO-controlled areas were adjacent to or overlapping those administered by the central government, there were often parallel systems serving the same or nearby communities. While those services offered by the government may

2 “Readout of the President’s Call with Daw Aung San Suu Kyi of Burma”, U.S. Embassy in Burma, April 6, 2016. [https://mm.usembassy.gov/readout-of-the-presidents-call-with-daw-aung-san-suu-kyi-of-burma/] 3 Notice of Funding Opportunity (NOFO) Number: RFA-482-17-000004, USAID Burma, March 14, 2017. 4 NOFO, op. cit. p.11. 5 Ibid. p. 5. 6 Ibid. p. 6. 7 Ibid. p. 10.


have been superior, the levels of distrust were such that ethnic communities would still opt for those provided by their own community groups.8

Conflict-affected areas often have poor access to healthcare, livelihood opportunities, and education services, especially in the most remote places. Perceptions of how services are provided and by whom are frequently tied to the political environment, serving to build or undermine local legitimacy and authority. For many people, ethnic or religious identity, mother tongue language, and historical alliances or fault lines are fundamental to choosing and trusting service providers.9

The ACE project was designed to build upon USAID’s previous assistance to communities along the Myanmar-Thailand border, following a number of guiding principles. It was designed to have a conflict- and context-sensitive approach assuming “continued momentum and progress on the peace process and democratic reforms.” It was to be flexible and adaptive to respond to local dynamics, take advantage of opportunities, and mitigate challenges. Using sub-grants as a critical tool, ACE was to deliver accountable, equitable, and transparent basic services “while modeling democratic principles of local governance and increasing the capacity of communities, civil society, and local decision-makers in conflict-affected areas to constructively engage with each other.”10

PROJECT GOAL AND OBJECTIVES

The ACE project contributes to the USAID mission objective “Strengthen institutions to promote good governance, rule of law and human rights, and to expand space for civil society and the media to support the democratic transition” and the DGHA office strategy “Democratic Governance Strengthened to Advance the Transition.”

ACE’s goal is that vulnerable communities will be empowered to engage in local decision-making processes. To achieve this, the project’s objectives and intermediate results (IR) are:

Objective I: Community Vulnerabilities Reduced:

IR 1.1 Health care access and coordinated service delivery improved

IR 1.2 Education access and services improved

IR 1.3 Sustainable livelihoods promoted

IR 1.4 Water, sanitation, and hygiene (WASH) improved

IR 1.5 Community capacity for preparedness, mitigation and response to disasters increased

Objective II: Community Participation Strengthened:

IR 2.1 Community capacity and leadership skills improved

IR 2.2 Participatory community decision making capacity improved

IR 2.3 Community action plans developed

8 See “Advancing Community Empowerment in Southeast Myanmar Baseline Assessment”, SoCha LLC, September 24, 2018, p. x. 9 See “Advancing Community Empowerment in Southeast Myanmar Baseline Assessment”, SoCha, LLC, September 24, 2018. 10 Ibid. pp. 19-20.


IR 2.4 Trust and reconciliation enhanced

IR 2.5 Youth and women’s participation increased

Objective III: Local Governance Systems Strengthened:

IR 3.1 Integrated public service provision increased

IR 3.2 Local public service provider capacity strengthened

IR 3.3 Trust and collaboration between communities and local authorities increased

THEORY OF CHANGE AND CRITICAL ASSUMPTIONS

ACE’s logic model (see Figure 1) is guided by its overarching theory of change:

IF community vulnerabilities are reduced, community participation is strengthened, and local governance systems are strengthened, THEN vulnerable communities will be empowered to engage in local decision-making processes.

ACE has a number of critical assumptions that underlie the project:

i) minimum political stability and engagement remains,

ii) access to villages is obtained,

iii) there are no overwhelming shifts in IDP and returnee movements,

iv) the government does not significantly withdraw services, and

v) no covariate shocks through natural disasters occur.

Figure 1. ACE Logic Model Y1–Y3


EVALUATION METHODS & LIMITATIONS CONTEXT

This evaluation was conducted during the 2020 COVID-19 pandemic when strict government restrictions on travel were in place, along with a government-issued stay-at-home order.11 In ACE’s operations area, EAOs enforced similar orders. For these reasons, the evaluation was conducted entirely remotely by team members based in Yangon and the Washington, D.C. area.

The circumstances affected the methodologies available to the ET. No direct observation of project activities was possible as travel to the project area was prohibited. Most physical project activities, such as in-person meetings and trainings, were suspended after government restrictions limiting public gatherings to less than five people. These restrictions also meant that focus group discussions were not an option as a research tool. In addition to a desk review of documents, the ET conducted key informant interviews online, and a mini-survey with direct beneficiaries over the phone.

DOCUMENT REVIEW

The evaluation began with a desk review of key project documents and, where relevant, some external research. During the key informant interviews (KIIs), additional relevant documents were identified and reviewed. Documents reviewed included (but were not limited to):

• ACE annual and quarterly reports • ACE monitoring, evaluation, and learning (MEL) plan • ACE baseline study • ACE Equity Inclusion & Conflict Sensitivity (EICS) strategy • ACE Mercy Corps Governance Analysis • ACE Applied Political Economy Analysis (APEA) • ACESM Conflict Sensitivity and Mitigation Framework • Identity Crisis: Ethnicity and Conflict in Myanmar, International Crisis Group • ACE Success Stories and Highlights (October 2020) • Shae Thot Final Report (June 2018) • ACESM Notice of Funding-Request for Application (NOF-RFA) (March 2017) • Stepping into Uncertainty: Refugee and IDP Experiences of Return in Southeast Myanmar, Karen

Human Rights Group (August 2020) • Livelihoods and Food Security Fund (LIFT), South East Myanmar Programme, Call for Concept

notes (November 2019) • Humanitarian Response Plan Myanmar, 2020 Programme Cycle, Office for the Coordination of

Humanitarian Affairs (OCHA) (December 2019)

A complete list of documents reviewed for this evaluation is in Annex II: Sources of Information. Most of the desk research was conducted before the KIIs or survey data collection.

11 See “Alert: Rangoon Region Stay-at-Home Orders”, US Embassy Burma, 21 September 2020 (Accessed at https://mm.usembassy.gov/alert-092120/ on November 30, 2020.)

https://mm.usembassy.gov/alert-092120/


KEY INFORMANT INTERVIEWS

KIIs are semi-structured interviews with individuals or small groups of people. For this evaluation, the ET conducted 43 KIIs with 44 people who are stakeholders in this project, with one interview being conducted with two people. From the original list of 46 proposed interviewees, three stakeholders were not available during the research period, including the two government officials proposed. This meant that there were no direct perspectives of government counterparts on the project. Each KII was, on average, about one hour long. The ET recorded and transcribed all the interviews, most of which were conducted in the Myanmar language and translated into English for analysis. In consultation with USAID/Burma and Pact, the Evaluation Team created the selection criteria for KIIs. These include those outside the project, such as donors and other implementers and at various levels within the project. Interviews were conducted with senior management in Yangon as well as field management and project implementation staff in the three regional offices in Hpa An, Mawlamyine, and Loikaw. ACE grantees providing support or services to multiple village communities through either the Critical Service Grant (CSG) or Responsive Health, Education, and Livelihood Services (RHELS) mechanism are another category. Village level grantees, such as Village Development Funds, who are also village level volunteers, were interviewed as part of the mini-survey discussed below.

The Evaluation Design and Workplan (EDW) listed 46 key stakeholders targeted for an interview by the evaluation team. As part of the evaluation design process, this list was prepared collaboratively with USAID/Burma’s M&E specialist, DGHA Agreement Officer’s Representative (AOR), Pact senior management, and other consortium members all having the opportunity to review it and suggest changes. Of these agreed stakeholders, only three were unavailable or unreachable during the research period. These included two government partners that were suggested by one consortium member. A full list of those interviewed is in Annex II: Sources of Information. The ET produced three interview guides for those external to the project, those working ACE, and those receiving grants (see Annex III: Data Collection Instruments). A full tabulated illustrative list of proposed interviewees can be found in Annex II: Sources of Information. A typology of KII interviews is below (Table 1).

Table 1. Typology Stakeholders Targeted for Key Informant Interviews

Stakeholder Type Organization/s Position/s

Donors & Other Implementers

(6 interviews)

USAID, OCHA (Myanmar Humanitarian Fund, MHF), & UNOPS (LIFT)

USAID/Burma DGHA Director, current & previous AORs; other donors & implementers working in southeastern Myanmar

ACE Consortium Partners

(26 interviews)

Pact, Save the Children International (SCI), Mercy Corps, and Community Partners International (CPI)

Senior in-country management as well as field managers & project implementation staff from each of the three regions

Project Advisors

(2 interviews)

Advisory Board members, who are independent consultants or working in the region for non-governmental organizations

International & national members


MINI-SURVEY

The evaluation team conducted a mini-survey as an efficient way to complement KIIs, gather perspectives of ACE’s direct beneficiaries, and analyze the effectiveness of the project at the village level. The mini-survey aimed to learn about village-level decision-making, including who is involved, and how they are implemented (EQ2). It was also designed to track the participation of women and youth and the effect of the COVID-19 pandemic on these village-level institutions (EQ5).

The mini-survey was of direct beneficiaries from the ACE project, defined as those who have received or have experienced some direct project intervention, such as village organizing or facilitation, training, mentoring, or coaching. These direct beneficiaries were volunteers taking part in an ACE-supported village level organization and they include members of Parent Teacher Associations (PTA), Village Coordination Groups (VCG), Village Development Committee (VDC), Village Health Committees (VHC), or Village WASH committees (VWCs). The mini-survey did not include participants of agriculture sector activities because a full list of such participants was not available ahead of the research period. For this reason, the mini-survey was not able to contribute to the ET’s understanding of agriculture activities. Villages that decided to engage with ACE also conducted Community Needs Assessments (CNA) as part of ACE’s overall Community Introductory Workshop (CIW) approach, after which communities decide together whether they wanted to participate in the ACE project.

SAMPLING

For the mini-survey, the ET created a stratified sample of the ACE project’s direct beneficiaries. Pact supplied a list of 2,317 village-level volunteers from the several types of organizations the ACE project works with at this level. This list was cleaned by removing those under 18-years-old,12 those noted to have no or unreliable phone connections, and those based in Thailand. This process reduced the total population of volunteers in the sample to 1,875 persons.

The sample drew from all four states or regions where ACE is active, all targeted townships, each sector where the project is working, and each type of committee it is interacting with at the village level. Within these committees, the mini-survey sample targeted one person in a leading role (such as the committee chair) and another member or volunteer who was not in a leading role. The ET selected respondents according to age (including youth 18-29 and adults 30 and older) if they were available. Of these, one female and one male were selected from each of the youth and adult groups. If there were more than one eligible person in any of these groups, one individual was selected randomly. The data supplied by the project had most but not all the information necessary for the sampling process. The list of Parent Teacher

12 Following Institutional Review Board (IRB) advice, the study team did not include youth under 18 years of age.

Grantees (Critical Service and RHELS Grantees)

(7 interviews)

A selection of CSGs and RHELS grantees, which sometimes overlap, will be interviewed

Director or senior manager from each grantee

Interest-Based Negotiation (IBN) Partners

(2 interviews)

A selection of two of the six IBN partners will be selected

Director or senior manager


Associations volunteers, for example, did not include age. For this reason, a screening question on age was administered as part of the survey. During the evaluation design, based on the data available, the ET estimated that the mini-survey would end with a sample size between 126 and 215 respondents.

ACE staff felt that suspicion could be caused if the survey enumerators “cold called” village-level volunteers. Therefore, direct beneficiaries selected to be interviewed were first contacted by an ACE staff member to alert them the evaluation team had chosen them for the survey. Potential respondents were also told that the interview was voluntary and informed them when to expect a Social Impact enumerator to call. The evaluation team prepared sample lists and a script for ACE staff to use to call ahead. Annex III: Data Collection Instruments contains the “call ahead” script and the draft mini-survey instrument.

The evaluation team fielded the survey between November 1-12 using five enumerators (four women and one man) fluent in the Burmese, Kayah, Mon, Poe Kayin, and Sakaw Kayin languages. Initially, the ET selected and attempted to call 215 respondents. When a respondent was unavailable, a new one, fitting the same profile (type of group, sex, and youth/non-youth status) was selected by the ET until this option was exhausted. The most common reasons for being unavailable was that the volunteer did not answer the phone or was away working in the field. Of the group of unsuccessful calls, 22 people (9 percent) refused to participate in the interview when spoken to by an enumerator. A total of 460 calls, including 274 replacement calls, were made, allowing the ET enumerators to complete a sample of 186 respondents.

The sample and its distribution by area, type, position, age, and sex are summarized in Table 2.

Table 2. Mini-Survey Sample Summary

Level Name/Category Distribution (percent)

State/Region Kayah 29 Kayin 37 Mon 18 Shan 16

Township Demoso 12 Hlaingbwe 13 Hpasawng 6 Hpruso 11

Kyainseikgyi 16 Pekon 16

Thanbyuzayat 18 Thandaunggyi 8

Committee Sector Education 16 Governance & Engagement 47

Health and WASH 23 WORTH 15

Position Level Leading Role (Chair/Head) 17 Supporting (Secretary/Notetaker) 13 Financial (Treasurer/Accountant) 24

PTA (Parent) 33

PTA (Teacher) 7

PTA (Moderator) 6

Age Youth 33 Adult 67


Level Name/Category Distribution (percent)

Sex Female 58 Male 42

Due to a longer than anticipated time to recruit, onboard, and train enumerators with the appropriate language skills, the ET administered the survey behind the original schedule. This delay meant the enumerators conducted interviews across the period of the national elections held on November 8, 2020. The enumerators interviewed more than 80 percent of the survey respondents before polling day, which was mostly peaceful and held without any significant incidents. The ET does not believe that the election influenced the survey's fielding nor the respondent’s answers. As respondents were all contacted by ACE staff before being interviewed, the evaluation team asked the enumerators to note at the end of each questionnaire any suspicions of respondents having used prepared answers. There were no reports of behavior consistent with the volunteers being coached ahead of time to answer the survey.

MINI-SURVEY CHALLENGES

The ET experienced challenges associated with conducting the mini-survey that could increase certain biases. Some participants showed a low level of interest in the survey. The selected respondents requested enumerators call back when they were less busy but then did not answer the calls. A few respondents said their family did not like being a committee member or that they were too new to the group to answer questions about its activities (although there was no requirement for length of membership). The absence of these least engaged and most negative groups could have resulted response bias. Mobile telephone coverage is not consistent in the project area, and many phones were out of range at the time enumerators called. This means the most remote and disadvantaged communities were under-represented. Poor connections periodically led to dropped calls, and some interviews required multiple calls to complete. Inferior quality calls combined with regional accents made answers hard to hear in some cases.


FINDINGS, CONCLUSIONS, & RECOMMENDATIONS

OVERVIEW

This section synthesizes what was learned during the evaluation, presenting this information as findings and conclusions organized by evaluation question. Question 6 asks for recommendation and therefore in this report, EQ6 is synonymous with the recommendations section. Findings are empirical facts based on data collected during the evaluation through the document review, key informant interviews with project stakeholders, or the mini-survey conducted with direct beneficiaries. The conclusions at the end of each EQ section synthesizes and interprets the findings and makes judgments supported by the findings. The recommendations section lists actions the evaluation team proposes for the consideration of program management based on the findings and conclusions.

EVALUATION QUESTION 1 — EXTERNALITIES

How have externalities related to the evolving political context in ACE implementation areas impacted ACE’s ability to achieve expected objectives over the first three years of the project?

The political environment at the national and sub-national level has changed since USAID first conceived the ACE project. The sense of optimism that greeted the new NLD-led administration in April 2016 has faded. With the lack of progress in the national peace process, there is a sense that expected political changes that could have positively impacted the southeast have not happened. With few changes to the conflict dynamics in the region, there have not been many noticeable peace dividends. Instead, since 2017, the number of incidences of political violence, riots, and protests has increased dramatically across the southeast, including political crackdowns on civil society organizations (CSOs) in Kayah State after protests against the building of statues of the first leader of the armed forces.13 In the design of this evaluation, the planned to examine how such events outside the project have impacted ACE’s implementation. The question for the evaluation is how these factors have impacted the project’s implementation and meeting its objectives by the end of Y3.

Political externalities impacting the ACE Project arise from the national, regional, and local levels. The Union level government controls the peace process. Relationships between and with the state governments, military, and EAOs set the political tone for the southeast as a region. In local communities, it is township-level governments that have the most impact on the project’s activities. At the national level, externalities of note include high-level perceptions of the NLD government and its ability to move beyond the NCA. Peace with the country’s ethnic groups was not made a national priority, and it was similarly unrealistic to expect that State governments would give it importance. Policy and guidance that could impact consistency in approaches throughout the country and in multiple states come from the national level. At the regional level, the mosaic of armed and political groups and their ongoing conflict with the military and government creates an operating environment with a high level of complexity where permission to travel and conduct activities can be unpredictable. At the local level, there is significant variation from one township to the next depending on the personality of appointed officials and their ability to operate in these areas where ambiguity remains the norm. This is where the key implementation relationships are found.

13 See Victoria Milko, “Ethnic minorities across Myanmar protest against Aung San statues”, Al Jazeera, March 29, 2019.


National After decades of military rule, the international community saw the installation in April 2016 of a civilian government led by democracy icon Aung San Suu Kyi as a cause of great optimism. Stakeholders working in Myanmar at the time described these as “heady days.” “Daw Suu (was) very adamant about the peace process being top of the agenda for NLD,” noted one international stakeholder, and how she wanted to tap into her father’s legacy “to reset the dialogue with ethnic groups.” The national view, particularly from those internationals sitting in Yangon, was mismatched with the more superficial change taking place. “The international perception was that we had a new government, but in reality, we had a continuation of the previous military government,” said a second international stakeholder interviewed by the ET.

When the NLD took power, optimism across the international community and the USAID mission saw the country on the cusp of momentous change. This was a view that resonated with the highest levels of the US government and was reflected in the design of the ACE project. USAID conceived of ACE when they expected that the NLD government coming to power would see a period of close cooperation between the new government, the international community, and civil society. “The optimism was valid, even if it was over rosy, but the complexity is enormous,” said a third international stakeholder. Instead, in the last five years, most international development partners have been frustrated by the first NLD government. There is now much distrust between the new administration and civil society, who international stakeholders interviewed said have been subject to more restrictions and greater scrutiny than under the Union Solidarity and Development Party (USDP). Anticipated democratic dividends have not been delivered. Since the design mission in late 2016, rather than opening up, stakeholders acknowledged Myanmar has defied expectations and the pace of reform has slowed down.

The pre-election rhetoric of Aung Sang Suu Kyi created elevated expectations that the peace process was at the top of the NLD’s agenda. The NCA was signed in October 2015 by the USDP administration under Thein Sein and inherited by the NLD. On paper, the NCA’s Article 25 says EAOs during the interim period would work together with the government on “projects concerning the health, education and socio-economic development of civilians.”14 “If you read this, you will think, wow, this is great,” said a fourth international stakeholder said. The ethnic service providers (ESPs) and the government were supposed to work together during the interim period, the same stakeholder added, however the peace process has been stalled since the end of 2018.15 Without effective progress at the national level, there was no coordination of policy about how lower levels of government should work with EAOs. This meant there was no consistent guidance that could have been enforced across all states and regions of the project’s area of operations in the Southeast. The net effect was that civil administrations in the states were left to determine their own approaches about how they related to EAOs and ESPs and those local and international NGOs who worked with them. At the same time, the more powerful military had a free hand to conduct operations and continue to disregard civilian authority.

With little change in the status quo of the peace process, only small numbers of refugees returned during the project period. From the refugees’ perspective, considerable obstacles to return remain, including concerns about safety, security, and having the means to sustain their livelihoods. Potential returnees worry about the challenges in obtaining identity documents that will unlock access to political rights, land,

14 See Article 25. A (1), The Nationwide Ceasefire Agreement between the Government of the Republic of the Union of Myanmar and the Ethnic Armed Organizations, 15 October 2015. 15 See Paul Keenan, “The Union Peace Accord: Moving forward after the election,” EBO Background Paper, No.6 2020, November 2020.


education, and employment.16 USAID designed the ACE project in response to these needs to prepare the sub-region for the possibility of larger numbers of returns at some time in the future.

But even before the Rohingya crisis of 2017, EAOs, ESPs, and CSOs in the southeast had observed the NLD was very top-down in its decision-making, and the peace process was not moving forward. For the international community, high-level project stakeholders recalled, it was a wake-up call that called into question future cooperation with the NLD government. After security forces attacked Rohingya communities in Rakhine State in August 2017, driving hundreds of thousands of them to take refuge in Bangladesh, attitudes were reinforced. For the ethnic communities, noted a fifth international stakeholder interviewed by the ET, “it was emblematic of what had been happening under the military for decades in Myanmar.” Stakeholders said the trust and confidence of the ethnic people towards the government, already at a low level, stayed low during the first NLD government and throughout the first three years of the ACE project.

Regional Some stakeholders refer to the southeast as a highly complex region. As ACE’s July 2018 APEA notes, even referring to “the southeast” may be contentious, “as it connotes homogeneity where there is none….The states and regions that share Myanmar’s eastern border with Thailand are vastly diverse, home to dozens of ethnolinguistic groups, various religions, several types of armed factions, and highly mobile or isolated populations that migrate for work or are displaced by conflict or large-scale development.”17 In the absence of a peace process and a common approach asserted by the national government on the region as a whole, power has defaulted to state governments and regional military commanders.

Fighting in the southeast continued, particularly between the Karen National Union (KNU) and the Tatmadaw, and the local communities were impacted by displacement. These tensions led the KNU to suspend its participation in key aspects of the NCA for a period in 2018-2019. With central and state government attitudes unchanged, new crises flared during the first three years of the ACE project. The building of a statue of General Aung San, the father of the NLD leader, by the Kayah State government in Loikaw in late 2018 led to rounds of protest by and repression against local ethnic civil society and political groups opposed to its construction. Insecurity caused by this event led to the temporary closing of project offices. Stakeholders told how these clashes increased distrust on both sides and after this confrontation the state government imposed new restrictions and required additional permissions for domestic and international NGOs. During the project period, stakeholders reported that periodic clashes in the region between the Tatmadaw and EAOs led to temporary travel restrictions by both sides restricting project activities.

In the absence of a strong centralized state, the region remains a complex political environment in which to implement. Stakeholders described a patchwork of areas controlled by the government, EAOs, or grey zones where both sides try to assert control. In these areas, communities can be subject to two sets of rules and even double taxation. Working to obtain the required permissions from the government and EAOs is time-consuming. Even after civilian government authorities or EAOs issue travel authorization documents, a trip may be thwarted when low-level military officials do not recognize them. Communities have low levels of trust in outsiders, including international non-governmental organizations (INGOs).

16 See “Stepping into Uncertainty: Refugee and IDP experiences of return in Southeast Myanmar”, Karen Human Rights Group, August 2020. 17 “Baseline Applied Political Economy Analysis of Southeastern Myanmar”, Pact, July 2018, p.2.


Building relationships takes time, and given these communities are in remote areas with poor communications, they are also time- and resource-consuming.

Local Stakeholders have observed that the working environment for the project is subject to highly localized variations. While Yangon-based stakeholders focus on what has not happened at the macro level of national politics, stakeholders living and working in the southeast have observed some progress amid the complexity at the micro-level of townships or village tracts. After the New Mon State Party (NMSP) signed the NCA in February 2018, stakeholders working in that region said communication between the government and local organizations became easier. Since the NLD came to power, some stakeholders have observed higher activism levels among CBOs, CSOs, especially women and youth groups. “Youth participation in politics has become significantly stronger than ever before. We also see political parties give new opportunities to young people and women,” said one grantee.

Implementation of policies often came down to individuals' personality and their ability to deal with ambiguity and risk. Stakeholders noted how township level administrators without experience working in conflict-affected areas could be reluctant to work with ESPs because of their association with EAOs. The 2017 National Health Plan recognizes EHOs as stakeholders in the formulation of Inclusive Township Health Plans (ITHPs) and should be represented on Township Health Working Groups.18 “In reality, their approach needs to be more systematic and institutionalized,” said one stakeholder working in the health sector. Even when high-level policy settings may be inclusive, supportive local officials are required to implement them. While another local staff member working in the health sector reported that “the government has become more willing to cooperate with EHOs in the states and regions”, senior project staff report that some EHOs find their government counterparts uncooperative and willing to encroach on their established areas.

Stakeholders describe significant variation working in townships due to the government personnel's qualities and their willingness to work with EAOs, ESPs, and INGOs. “We don’t get along well with every general administrator in each township,” observed one ESP stakeholder. “Some are friendly and open-minded, and some aren’t.” Staff turnover due to natural bureaucratic cycles can dramatically impact the project’s ability to implement its activities. Stakeholders acknowledge this is not just a challenge to projects in the southeast but a common challenge for development practitioners in Myanmar. The rotation of a township level administrator with a good relationship with the project and its partners can slow down activities. “You have to re-explain the whole program to somebody new who might be a bit more skeptical, which is independent of the broad dynamics at play,” said one stakeholder. A fresh round of introductions, orientation, and explanations are required. Stakeholders noted that approvals are not immediate and require the repeated expenditure of time and effort. ACE and USAID have recognized the challenges in overcoming this distrust and lack of progress in bringing the two sides together. Stakeholders said the project no longer talks about the “convergence” of government and ethnic service providers. High-level stakeholders said the political externalities are not conducive to such a goal. In the Y4 MEL Plan, USAID and ACE acknowledged these reduced expectations by agreeing to revise IR III.1 from “integrated service provision increased” to the more modest “coordinated service provision increased”.

18 See “Highlights from the Annual Operational Plan 2017-2018”, Ministry of Health and Sports, Republic of the Union of Myanmar, 31 March 2017, p.2.


CONCLUSIONS National

• Lack of progress in the peace process by the NLD government has meant an absence of proactive national level policy that could have been asserted from the center and brought a consistency of governance to the entire Southeast region.

• Thus, control of the political context in the Southeast was left to regional powerholders, such as state governments and the military.

• Political externalities at the national level were more complex than what ACE would have faced had peace prevailed; instead, confrontations continued between the government and EAOs, including military clashes and protests, causing ongoing security challenges for the project, partners, and communities.

Regional • In the absence of clear policies to promote peace and inclusion of ethnic minorities as national

priority, ACE and its partners were often stuck in a web of state-level bureaucracy, when permissions were denied, and each roadblock required additional time to appeal and ultimately resolve.

• The project’s first two objectives of strengthening community vulnerabilities and community participation could still be advanced but implementation was slowed down by these regional political externalities.

• The third objective of strengthening local governance systems was most affected by ongoing conflict only heightened the long-standing distrust between government and ESPs sides that undermined ACE’s intended purpose of bringing them together.

Local • Inconsistent national and regional approaches and ongoing regional conflict created uncertainty at

the township level. In the absence of clear direction and guidance from the national government, relationships between local administrators and their ethnic counterparts were determined as much by personal discretion as official policy.

• Poor local relationships between township administrators and government and ethnic service providers affected the project’s third objective as convergence and other possible cooperation and synergies remained unrealized.

• This in turn impacted the project’s first objective of reducing community vulnerabilities as it meant that project beneficiaries, especially in non-government-controlled areas, did not receive as full a range of often higher quality government services that villagers were entitled to and that might have benefited them.

EVALUATION QUESTION 2 — PROJECT ACHIEVEMENTS

To what extent have ACE’s health, education, livelihoods, water, sanitation, and hygiene (WASH), and disaster risk reduction (DRR) interventions contributed to achieving its expected outcomes, intermediate results, and objectives at this point in the project implementation?

ACE’s overarching theory of change is that if i) community vulnerabilities are reduced, ii) community participation is strengthened, and iii) local governance systems are strengthened, then vulnerable communities will be empowered to engage in local decision-making processes.

The critical assumptions that underlie the project were that i) minimum political stability and engagement remains, ii) access to villages is obtained, iii) there are no overwhelming shifts in IDP and returnee


movements, iv) the government does not significantly withdraw services, and v) no covariate shocks through natural disasters occur.

These assumptions have generally held during the project’s first three years. As noted above in EQ1, the political context was challenging for the project, but this was not entirely unanticipated. While Myanmar was generally politically stable during the first three years of the project period, there was some localized instability in the project area due to armed clashes and political conflict between regional governments and ethnic groups. Access to village was sometimes restricted because of this conflict by either the government, the military or EAOs. At other times, historic distrust of outsiders by EAOs was an impediment to access. Since Q3 in Y3, pandemic restrictions have been another factor that has cut off the project’s access to many villages and postponed activities. While the armed clashes created some internal displacement in the project area, the lack of process on the peace process discouraged large scale return of refugees. However, the pandemic has increased the rate of return of migrant laborers, although reasons for return are often complex.19

While this EQ is framed using the five thematic pillars, the governance work of ACE is cross-cutting as it frames the support given by ACE to the village in the sectors of education, health and WASH, and livelihoods. The project’s governance is centered on the VCGs/VDCs and starts with a community introductory workshop (CIW) process, which is the first step of engagement in Pact’s direct implementation activities. During the first three years of ACE, 447 CIWs were held with 367 villages agreeing to participate in ACE’s community-based engagement approach, which meant either a VCG or VDC was revitalized or started. By the end of this period, as record in the Y3 annual report, Pact had supported 390 villages through such groups to develop Community Action Plans (CAPs) and implement 703 actions based on these plans. This is contributed directly to the project’s intended result 11.2 and 11.3 that seeks to improve participatory decision-making and develop CAPs. The additional insight that the ET contributes is that these groups are also contributing to IR 11.5 to increase youth and women’s participation, not only in the numbers present, but also how they participate, which will be explained below. The mini-survey interviewed 87 respondents who were members of governance and community engagement (GCE) groups, like VCGs and VDCs, and asked how often they participated in their meetings. Women and men in GCEs appeared to actively participate almost equally in these meetings (Figure 2), unlike other groups such as Health and WASH committees, which staff member stakeholders who work with them say are often more dominated by men.

19 See “Understanding the Experiences of Returning Migrants Affected by the COVID-19 Pandemic in Myanmar”, IOM, October 2020.


Figure 2. Participation by Sex in GCE Groups

EQ2.A: HEALTH CARE ACCESS AND COORDINATED SERVICE DELIVERY IMPROVED

ACE provides support to health care services in the Southeast in three ways. First, it seeks to contribute to better health outcomes (Objective I). Second, it tries to improve the way decisions about health care are made and widen the circles of those involved in making them to include more women and youth (Objective II). Third, the project seeks to support systemic change by improving the way government and EHO health care systems are managed and the two systems interact (Objective III).

In its Y3 Annual Report, ACE reported its life of project (LOP) health outcomes for Objective I of reducing community vulnerabilities. Comparing to the project’s baseline, ACE recorded improvements in the number of births that were attended by skilled workers, pregnant mothers who received antenatal care services, and children under five that were vaccinated and treated for diarrhea. However, ACE did not meet all targets for improvements and some activities appear to have been disrupted more by the pandemic restrictions or were harder to achieve than others. ACE achieved 84 percent of the target for birth attendance but only 56 percent of the target for treating cases of childhood diarrhea. By its own indicators, the project has been least successful in improving access to quality health services, achieving 33 percent of its target.

The mini-survey conducted by the ET provided an indication of how the project is making a contribution to its Objective III of strengthening community participation. In its Y3 Annual Report, ACE reported helping form 228 VHCs and training 303 village health volunteers. Before these committees were formed, health and WASH decisions were not made collectively. Given these groups are an output of the project, the insight from the mini-survey is that the health and WASH committees are contributing to decisions being made in a more democratic and inclusive way. This is a positive outcome for ACE because traditionally such decisions have been made by men or one person.

The mini-survey recorded significant levels of participation by women and youth in health and WASH meetings. For example, equal numbers of men and women (10 respondents) report participating in discussions “very often” (45 percent) (see Figure 3). In the mini-survey sample, 43 of the 186 respondents were members of a health or WASH group. More than 60 percent of these groups (26) had a female membership of at least 50 percent or above. Women reported a high level of participation in the meetings with 68 percent of them participating very often or often in the meetings (see Figure 3). Almost 45 percent of these groups (19) had more than more than 50 percent youth participants (18-29 years). Respondents


in health and WASH groups said 86 percent of decisions were made by consensus and of these decisions all (14 percent), almost all (35 percent), or a few (9 percent) were implemented. However, like all village-level organizations, the restrictions of the COVID-19 pandemic have impacted their efficacy. Since the pandemic began, more than half (23 groups) responded that they have not met.

Figure 3. Women’s Participation in Health & WASH Committees

Working with local partners, ACE has also made some progress in its Objective III of strengthening local governance systems. The project’s work in this sector is often implemented through local partners receiving CSG or RHELS grants, including ESPs that are the local government providers in areas controlled by EAOs. Delays of up to two and a half years in issuing sub-grants, which were reported in some interviews with key informants, were reported to have translated to slow implementation and the project under-performing against some of its targets. The most difficult part of working towards this objective has been bringing together government and ethnic health service providers. Several stakeholders highlight the “trust pilot” program in Pekon Township Health Department and Civil Health and Development Network (CHDN) as one positive example of the way the governance of health services might change.20 This pilot, which is a collaboration between Pact, CPI, and CHDN, did have buy-in from stakeholders and was showing signs of having great potential. Stakeholders involved with pilot said it was possible because of good relationships on both sides in this location. This would have ended decades of non-communication between Township Health Departments (THD) and EHOs. Unfortunately, as noted in the Y3 Annual Report, coordination meetings, originally scheduled in March, were postponed due to the COVID-19.21

EQ2.B: EDUCATION ACCESS AND SERVICES IMPROVED

ACE provides support to education access and services in the Southeast not only to contribute to education outcomes and numbers of Parent Teacher Associations engaged (Objective I) but also to improve the capacity and leadership of these PTAs (Objective III).

In working to achieve its Objective I of reducing community vulnerabilities the ACE project has sought to improve education access and services (IR 1.2). ACE provides a broad range of support to government schools through SCI and through its local grantees to Ethnic Basic Education Providers that together

20 See “Success stories & Highlights (EOW), ACE, October 2020, p.77. 21 See “Year Three Annual Report”, ACE, November 14, 2020, p.22.


contribute to improved education access and services result areas. This support includes teacher training, curriculum support, providing educational materials and other learning tools, student support groups, and helping with school governance through training for Parent Teacher Associations. In the Y3 Annual Report, ACE notes it supported 374 PTAs and trained 2,712 teachers and PTA members, with 1,747 parents participating in education awareness sessions and coordination meetings. The project was most successful in reaching its target of reaching learners with educational assistance, with 54,416 primary students receiving assistance by the end of Y3, which was 81 percent of the target. ACE was less successful in engaging the targeted numbers of PTAs, reaching only 374, which was 51 percent of the target. The ET understands that the missed target was a result of school closure due to pandemic restrictions.

The mini-survey gives an indication of progress in the education sector towards Objective II and community participation strengthened by involving more women and youth. It should be acknowledged that this data gives only a partial picture of the project as the sample included only PTA members in government schools. However, it can provide some insights into the potential of this work to improve women’s participation. Of the 29 respondents interviewed who were active in PTAs, 20 respondents (69 percent) reported that their group had a majority of female members and nine said they also had a majority of youth members (under 29 years old). The mini-survey appears to confirm active participation of women in these groups. Most women (14 of 16) said they actively participated in PTAs (see Figure 4). Members reported that 90 percent of decisions were made by consensus with all (10 percent) or almost all (62 percent) being implemented. “In the past, PTA decisions were made by the head teacher,” noted one stakeholder working in this sector. “Parents did not have any role in decision making,” one project stakeholder said, adding that both parents and teachers welcomed the innovative approach.

Figure 4. Level of Women's Participation in PTAs

In terms of the project’s education work contributing to Objective III in strengthening local government systems, the ET was not able to provide significant insights due to not being able to gather data directly from either government or ethnic education providers.

EQ2.C: SUSTAINABLE LIVELIHOODS PROMOTED

ACE supported sustainable livelihoods by improving access to finance and training famers to use new techniques and practices (Objective I) and providing women with access to economic resources (Objective II).

ACE’s provided training on good agricultural practices and fund management to farmers and those in agricultural value chains. Stakeholders working on agriculture who were interviewed perceived ACE’s


work in this area to be the weakest and most problematic of the project. These stakeholders characterized progress in agriculture as localized rather than systemic or sustainable. In terms of its Objective I indicator, ACE had only reached 35 percent of its end of project target of providing assistance on new technology and management practices to 7,800 farmers by the end of Y3.

Stakeholders when interviewed had various explanations for this underperformance. There appears to be a mismatch of expectations within the project about what ACE could achieve in the agricultural section and part of the problem appears to lie in an apparent misunderstanding about the limits of the ACE mechanism. The Shae Thot project implemented by Pact and other partners in the dry zone between 2011-2018 was referred to by many stakeholders as the predecessor to ACE. Cesvi, a partner in that project, provided several types of agricultural training alongside distribution of agricultural inputs such as seeds, fertilizer, and technology.22 In contrast, while the ACE’s interventions are limited to providing training and guidance without distributing agricultural commodities, pesticides, and fertilizers, which are subject to statutory constraints.23 Some project stakeholders blamed the lack of such inputs as one reason for slow progress in the agricultural sector.

USAID’s perspective is that agricultural inputs were never conceived as part of the ACE project as it was not planned as an agricultural project.24 ACE was not designed to operate like either Shae Thot, which was implemented by Pact and other partners in the dry zone, or the Project for Local Empowerment (PLE), implemented between 2011-2017 in the southeast by a consortium headed by World Education. However, for village level organizers, providing farmers with a tangible benefit, such as agricultural inputs, is seen as a pre-requisite to get their buy-in to training programs. “You do need to have something you are bringing in, and something you are providing, beyond training, and a few kits,” observed one stakeholder working in agriculture. Without such incentives, the motivation to participate in ACE’s training activities is low, the same stakeholder reported.

USAID is considering an amendment to ACE’s IEE to allow use of pesticides, fertilizer, and irrigation. Stakeholders at the field level tended to blame these USAID restrictions for inferior performance. “Because the project is restricted from giving such material support, there is a struggle to achieve effective outcomes,” said one implementation-level stakeholder. Management-level stakeholders with an overview of the project acknowledge that the original design did not plan or anticipate the permissions required to provide such inputs into the agricultural sector and the time seeking waivers would take. USAID said that seeking waivers to overcome these constraints after the project begun required a level of proactivity and planning from the consortium management that only came later in the project. USAID was said to have advised ACE against requesting these agricultural inputs waivers due to the relatively small amounts involved, long chain of internal approvals needed, and the potential that USAID management could reject them.

The WORTH women’s empowerment groups were seen as a sustained achievement of Shae Thot and this “award-winning savings-based women’s economic empowerment model” was replicated in ACE.25 To the end of Y3, some 1,445 women had been involved in WORTH groups and improved their access to finance. ACE sees WORTH as a success story of its project and a route to “empowerment without dependency” for village women.26. However, progress with the relevant Objective I indicator also shows

22 See “Shae Thot Final Performance Report”, Pact, June 2018, p.56 23 Ibid, p.60. 24 Ibid, p.60. 25 “Success Stories”, op. cit. p. 72. 26 Ibid. p. 66.


that project underperforming in providing individuals with improved access to finance and by the end of Y3 ACE had only reached 21 percent of its end of project target.

The ET’s mini-survey included 27 respondents who were members of WORTH groups (25 women and 2 men; 12.5:1 ratio). Female members surveyed reported elevated levels of participation in the meetings when most participants were women, with 74 percent of female members participating very often or often (36 percent). Decisions are almost always made by consensus (93 percent) and always or almost always implemented 59 percent of the time. The mini-survey as a quantitative tool did not collect information on why decisions were not implemented. It did show that WORTH groups to be the least disrupted of all village-level organizations during the COVID-19 pandemic with only 15 percent of them having never met since March, compared with 32 percent of GCE groups, 53 percent of Health and WASH committees, and 59 percent of PTAs. As per Figure 5, WORTH groups have been able to maintain their momentum throughout the pandemic. It is unclear whether this is due to the nature of WORTH groups being important during an economic crisis, its sex composition promoting greater cohesion and solidarity, or other factors.

Figure 5. Impact of COVID on WORTH Group Meetings

In terms of Objective III contributions in the livelihoods area, the ET is limited in its ability to make observations, in part due to unsuccessful attempts to interview a government official working on agricultural issues during the research period.

EQ2.D: WATER, SANITATION, AND HYGIENE (WASH) IMPROVED

ACE improved water, sanitation, and hygiene by training individuals on water security, integrated water resource management, and/or water source protection and improving their access to sanitation facilities. (Objective I). The project strengthened community participation by organizing WASH committees and training community volunteers (Objective II).

By its own measures, ACE has had a high-level of achievement of its Objective I results in the WASH sector. As reported in the Y3 Annual Report, it has already met 96 percent of its end of project target in terms of educating people how to use new tools, approaches, and methods for water security, integrated water resource management, and water source protection. It has already reached 85 percent of its target of providing 68 percent of households in its project villages with improved access to sanitation facilities.


There was mixed data on how Health and WASH activities have contributed to the project’s Objective II of strengthening community participation. Health and WASH groups were among those volunteer organizations most impacted by the COVID-19 pandemic. According to the mini-survey, more than half of Health and WASH groups (54 percent) had stopped meeting during the COVID-19 pandemic (Figure 6).

Figure 6. Impact of COVID on Health & WASH Group Meetings

While noted above that these groups had elevated levels of female and youth membership and participation, one stakeholder working at the implementation level with WASH committees observed these groups had a problem with inclusivity. “The involvement of youths and women is weak…. Mostly decisions makers are men,” one project stakeholder said, referring to the activities of these groups. While the numbers in the mini-survey survey are small, the trend on participation is clear. Health and WASH groups, which the qualitative research said were male dominated, showed the same phenomena in the quantitative research. Within the Health and WASH groups, the highest number of women reported taking part in discussions during meetings never (3 out of 22 or 14 percent) or sometimes (4 out of 22 or 18 percent) (Figure 7). By comparison, women participants in the female dominated WORTH groups took part in discussions either very often (68 percent) or often (32 percent).


Figure 7. Participation of Women in Health and WASH Groups

The volunteer data supplied by ACE to the ET used mixed terms to classify members of these groups as either health, WASH, or health and WASH volunteers. Because health and WASH groups were combined for the mini-survey, this data cannot be disaggregated to examine any differences in participation between the two groups. The project may want to examine this issue further to better understand this dynamic and its implications.

Some stakeholders said they believed that WASH groups would have been more successful if USAID restrictions on small scale infrastructure did not apply and the project had been able to support the building of latrines more directly by providing materials. These same stakeholders cited Shae Thot as a model project, during which communities working that project constructed or renovated more than 54,000 fly-proof latrines, even though 74 percent of those were constructed or renovated by community themselves. 27 The Y3 ACE Annual Report notes that as part of its training for 179 WASH volunteers in 105 villages Pact disseminated information encouraging household use of fly-proof latrines.28 VWC activities, such as improving sanitation around village wells, have been carried out using Village Development Fund (VDF) resources.29 Small scale construction is unallowable under ACE’s agreement and project stakeholders said the VDF funds that supported these initiatives did not include latrine construction.

The ET was not able to gather significant data on how Health and WASH activities contributed to Objective III of strengthening local governance systems.

EQ2.E: COMMUNITY CAPACITY FOR PREPAREDNESS, MITIGATION AND RESPONSE TO DISASTERS INCREASED

ACE’s work to train communities in the southeast in disaster preparedness and increase the percentage of targeted communities with a disaster preparedness plan was planned as Y3 activity. While there have been some preliminary activities, there was at the end of Y3 no data to report against the project’s performance indicators. These activities had started in Y2 Q3 when consortium partner Mercy Corps hired a DRR coordinator. As noted in the Y3 annual report, the DDR activities had begun with a series

27 “Shae Thot Final Performance Report”, op. cit. p. 43. 28 “Y3 Annual Report”, op. cit. p. 15. 29 “ACE Success Stories”, op. cit. p.89.


of meetings with CSOs, INGOs and government actors designed to gather information about disaster-prone areas in the region. Three CSO partners were selected to conduct community-led risk and resilience assessments and Hlaingbwe was identified as the most disaster-prone township. “It had barely started and was supposed to take-off in Y3, when it became a victim of COVID-19”, noted one stakeholder. These activities, which require face-to-face interactions with affected communities, are now planned in Y4, another stakeholder noted. For these reasons, it was not possible to assess how the project’s DRR activities had contributed to its objectives.

While not explicitly part of this intended result, the project was able to take humanitarian action and provide some disaster relief to remote and flood-stricken villages in 2018 and 2020. Assistance was coordinated with the Department of Disaster Management and KDHW, an ACE local partner, and included distributing hygiene packs to vulnerable communities.30 Seasonal flooding remains a perennial challenge for the southeast where it hampers efforts to sustain agricultural production and complicates managing hygiene during the pandemic. The need for DRR is apparent, but the ACE intervention was planned to take place over an extended period in years 3 to 5 of the project. “It needs to be sustainable and communities will be stronger if they have time to build relationships with government departments,” said one stakeholder. In dollars terms, this component is the smallest part of ACE’s work.

CONCLUSIONS • Four of ACE’s five critical assumptions have been challenged in the first three years of the project,

each having some impact on project performance, primarily by slowing down implementation or forcing the postponement of activities, most dramatically in the disaster reduction component.

• Before the pandemic disrupted ACE activities across the board, the project was showing progress towards its Objective I of reducing community vulnerabilities. Project reporting showed some achievement of intended results with health, education, & WASH improving, while sustainable livelihoods were being promoted.

• Pandemic restrictions have curtailed ACE activities across the board, especially the component focused on community preparedness for disasters, which was planned to start in Y3.

• While the ET was not able to gain fresh insights into all the project’s Objective II results, the project reports that communities are being trained, developing action plans, and implementing them. There is some evidence from the evaluation’s mini-survey that the way decisions are being made is sometimes more democratic and that women and youth are able to participate in these discussions and decision-making, especially in groups where they are well-represented such as WORTH groups and PTAs.

• With trust between government and ethnic organizations still low and often challenged by regional political tensions, ACE has struggled most with its Objective III aimed at strengthening local governance systems.

EVALUATION QUESTION 3 — PROJECT APPROACH

To what extent has a multi-sectoral and integrated community development approach contributed to achieving expected objectives, intended results, and outcomes in the areas of: a) Reducing Community Vulnerabilities; b) Strengthening Community Participation; and c) Strengthening Local Governance Systems?

With its multi-sectoral and integrated approaches, ACE as a project aspires to make a contribution that is greater than the sum of its individual partners or thematic activities. The ET examined the project at

30 Ibid., p.88.


three levels. First, it looked at whether there is a multi-sectoral approach when consortium partners implement their respective sector activities in one field location. Second, it examined the village-level integrated community development approach used by Pact in its direction implementation in a limited number of communities. Third, it reviewed how the project brings different partners and sectors together, especially with government.

In analyzing interview responses, the ET found a misunderstanding of the terms “multi-sectoral” or “integrated community development” approaches across the ACE consortium. Neither term appears in its annual reports. When they were used as part of this evaluation, the terms elicited different responses in KIIs from stakeholders from Pact, consortium members, and local partners.

While Pact uses the terms “multi-sectoral” and “integrated community development” in its reporting to describe its village-level programming, when the ET interviewed its staff members, these terms did not appear to be common usage. In answering the EQ3 questions in the KII instrument, only one of 11 Pact field staff distinguished between the two different models it uses, which are Integrated Services Model (ISM) and Community Leadership Model (CLM). In retrospect, these terms, and the way they are used (or not used) in this question may have caused some confusion. Staff also understood the project’s “multi-sector approach” as an explanation for its working with women and youth, which was addressed above in EQ2, and sharing offices with consortium partners, which will be addressed below in EQ4.

ACE was inspired by the work that Pact’s predecessor project Shae Thot did in Myanmar’s Dry Zone between 2011-2018. Shae Thot was based around the four pillars of agriculture, health, WASH, and Women’s Economic Empowerment. The consortium in that project adopted a “One Team” strategy for implementation by the project consortium, which, the project noted, “where relevant, the partners introduced and implemented the program together.”

ACE was not conceived in this same way. Co-implementation at the village level was considered to be the exception rather than the rule. Geography made this “one team” approach difficult. “We are in different areas, there is no multi-sectoral approach, it cannot be implemented,” observed one consortium stakeholder. The partners came together to respond to the RFA based on their existing experience and relationships. These were complimentary and it ultimately made them a successful bidder. Pact had experience with this type of programming, but mostly in the Dry Zone. SCI was working with government schools in the southeast. CPI had long-standing relationships with EHOs in non-government-controlled areas in the region. MC had extensive experience promoting peace and governance in Kayah State, as well expertise in enhancing livelihoods through working with private sector actors in agricultural supply chains.

The different expertise of each consortium partner naturally led them to focus on their strengths. Once the agreement was awarded, the consortium partners gravitated towards the types of activities they knew best and worked with local partners and communities where they had pre-existing relationships, resulting in limited geographic overlap. For example, SCI works with schools in government-controlled areas, which happen to overlap in 69 of the 367 villages where Pact does direct implementation. The consortium multi-sector approach allows ACE to cover extensive ground, especially when local grantees are engaged. Through all its consortium partners, the project’s footprint is extensive. Collectively, ACE works in some 29 different townships and 1,451 villages.

There are structural reasons that separated the work of the consortium partners. As is common practice for INGOs in Myanmar, each consortium member works under different government MOUs. While ACE submits and USAID approves one workplan for the consortium, member organizations reported having their own workplans as ACE is often only one project activity conducted by the partner. Some activities were funded by multiple donors. Coordination was possible at the consortium level but integration at implementation level was often not practical. “The intervention was changed from an integrated to a cooperative approach,” noted one consortium staff member. When answering this question, stakeholders,


particularly those working in the three regional offices, often referred to the benefits of having a shared workspace, which will be discussed in EQ4 below.

An integrated approach was easier for partners to implement within their own project activities. Pact drew on its experience with Shae Thot to guide its own implementation at the village level in ACE. There was a direct link and some overlap between the two projects. At the inception of the ACE project, 20 villages from Shae Thot project were identified as potential villages to continue participation in the ACE project, although Pact was able to begin working with only seven villages to KNPP restrictions on development activity in Kayah State. The Shae Thot model was compelling, some ACE stakeholders noted, and an “integrated approach” was an important part of this previous project’s approach. It was hoped that this success in Shae Thot could be replicated in ACE. In describing Shae Thot, it’s partners used the phrase “1+1=3” to describe this additional impact. The best description of this “integrated approach” is found in the Shae Thot final performance report:

“Shae Thot defined the project’s integrated approach as governance, health, WASH, livelihoods and food security integrated with the goal of creating durable village-managed funds developing the capabilities of individuals, communities, local organizations, and national institutions; and improving access to key products, services and information, as well increasing household income.”31

In ACE, Pact started with the integrated approach that was built around the Village Development Committees (VDCs), also known as the Village Coordination Group (VCG). Under ACE, it sought to integrate Village Development Funds (VDFs) into to the VDCs/VCGs.32 The Integrated Services Model (ISM) approach provided humanitarian support simultaneously in the sectors of maternal and child health; livelihoods and food security; and WASH. The project aims to promote integration through the VDCs by helping them develop Community Action Plans (CAP) to bring together in a coordinated way the support to different sectors that they receive. According to several interviewees, adopting such an approach had merit. “It is important to be responsive to what communities need,” observed a different stakeholder. “There isn’t a single need in any of these communities that can be prioritized,” they added. This approach worked in Shae Thot and was brought to ACE. “There was a real feeling that the integrated approach was a bit like a magic elixir that could somehow be transported to this new context,” observed one management-level stakeholder. “People were not necessarily naïve about how difficult that would be, but I think in practice it has been a lot more complicated than the initial design might have suggested,” they added.

One challenge was that Pact soon found that the Integrated Services Model (ISM) could be as much a burden as a blessing for these villages. “Our process is good, but the community cannot give us the time,” said one staff member. The Southeast is a significant source of migrant labor and many working age adults had left, leaving behind depopulated villages of the very young and the old. The levels of education in southeastern villages were also low, including fluency in the national language. This meant often one or two more educated people were disproportionately leaned on as contact persons with outside organizations and there was a small pool of potential committee members to serve on new village governance structures. Stakeholders saw this as an important contextual difference from the Dry Zone. “The villages have to depend on only one or a few leaders,” said one stakeholder. Forming multiple committees for health and WASH, education, and livelihoods meant overwhelming community partners who were needed for the success of the component of the ISM.

31 Ibid. 32 See. ACE YI Annual Report 2018, p. 38.


There was a recognition that ACE may have been too ambitious in not only its expected level of engagement but also its geographic scope. Implementing ISM in a village was an intensive process over three years. Pact’s original idea of directly community-based direct implementation in 15 townships was quickly scaled back to eight. The ISM delivered the integration of multiple sectors at the village level. It offered a menu of options covering health, livelihoods and WASH and enabled villages to select which components they wanted based on their village needs. Stakeholders said it was revised after it was found “too cumbersome an approach.”

Pact then adapted to the “Community Leadership Model” (CLM), also known as Governance and Community Engagement (GCE), which is offered to all villages. The GCE consists of support for a Village Development Fund (VDF) as well as the revitalization or creation of a VCG/VDC. CLM focuses on building the capacity of communities in governance and leadership and integrating one other sector, either livelihoods, health and WASH, or education. The project planned to implement the CLM over two and a half years in communities close to villages participating in the ISM pilot. According to the Y3 Annual Report, Pact continues to implement these two designs in the 367 villages in the four states and eight townships where it does direct implementation. As of the end of Y3, ACE through Pact was implementing the Integrated Service Model in 113 villages and Community Leadership Model, sometimes known as GCE plus 1, in 254 villages.33 In Y3, Pact reached its target maximum number of villages.

The final part to this question is how ACE with its different sectoral approaches, including those implemented by local partners, have been brought together with local government. The project does try to use its community engagement at the village level to bridge divides. “When we facilitate trainings, we invite government officials to the village, so they can meet and start relationships,” said one Pact stakeholder. These are tentative steps to build confidence with room for growth in an environment where there are long-standing trust issues between ethnic communities and government. But as noted above in EQ2, the project has been most successful in bringing ESP together with government in the health sector, albeit with modest results. This work is led by CPI and involves ACE EHO partners was showing some promise, but these efforts have been stalled by pandemic.

What is missing from the understanding of this evaluation question is the perspective of village level direct beneficiaries on these two Pact approaches. With the remote evaluation methodology, it is hard to understand how these the ISM and CLM approaches are working in practice and what impact they have had on the project’s objective. The quick telephone survey, which is a short quantitative tool, was not well-suited to build the trust and rapport needed to explore villages experiences with these different approaches. For example, it difficult to measure using this tool how well the consortium partners work together in the 69 villages where Pact and SCI do implement side by side. KIIs with key stakeholders was too subjective a tool for such an analysis. Nor was the short mini-survey a suitable tool to explore the complex and maybe subtle differences between the integrated services and community leadership models. A combination of direct observation and focus group discussions may have been better suited for such a task.

CONCLUSIONS • While each consortium partner contributes to ACE’s Objective I of reducing community

vulnerabilities by directly implementing their own activities, there is not a common understanding within the ACE of what it means to take a “multi-sector approach”, leaving an impression of a weak common purpose across the project, and few examples of how the project’s outputs are greater than the sum of its inputs.

33 “Y3 Annual Report”, op.cit., p.6.


• Pact’s implementation of two different models of village development both demonstrate progress in contributing towards ACE’s Objective II of strengthening community participation by changes in the way decisions are made and who makes them. The evaluation data did not point to the relative effectiveness of one model over another. Further study would be required to properly answer this question.

• The multi-sectoral potential that ACE offered seems to be least effective in Objective III of strengthening local governance as the opportunities for cooperation across sectors, for example between health and education, are weak as government is siloed, trust between ESPs and government remains low, and progress on bringing ethnic communities together with the government stalled by pandemic restrictions. While similar challenges were present in Shae Thot, stakeholders assessed the implementation environment for ACE to be less favorable.

EVALUATION QUESTION 4 — PROJECT STRUCTURE

How has the operational structure of the ACE activity (a combination of direct implementation, consortium partners, and sub-partner grants) affected ACE’s ability to achieve desired outcomes and results at this point in the project implementation?

ACE has a complex and multi-layered structure to implement its activities. Did this approach contribute to or hindered the projects results?

EQ4.A: FROM THE PERSPECTIVE OF THE PRIME IMPLEMENTING PARTNER

As a large multi-sectoral project covering a wide region, the consortium model with a prime leading consortium partners and guiding sub-grantees is seen from the top as a good model. Pact brought to ACE its experience managing a similarly large and complex consortium in Shae Thot in Myanmar’s Dry Zone. “Based on the design, being multi-sectoral, this is really the only way you could organize it. I can’t imagine any other way,” said one Pact manager. “It really benefits from having strong international partners who are experts in their field and who know Myanmar very well and who can come in and provide really useful technical support,” they added. Stakeholders say it has the potential to bring together partners who have specialty expertise and workstreams that otherwise would not coordinate or connect with each other. The consortium structure can prevent duplication and aid communications, some say. Some Pact field staff said direct implementation was an effective way to deliver benefits to the community. From the inside, with a high-level of centralization in its structure, there can be “great connections and synergies” under good leadership and management, one stakeholder observed.

Pact has considerable experience directly implementing in Myanmar. Within the prime, stakeholders had differing views about whether the right balance had been found in ACE between the prime’s own direct implementation and working through local partners. For some Pact stakeholders, direct implementation was effective for the community as it is a way to increase participation, performance, and management of activities. For other prime staff members, the bureaucratic obstacles created by government authorities in the Southeast showed its disadvantages. Restrictions imposed after the pandemic have made some in the prime reflect even more on whether there was another approach. “There was a lot of energy around direct implementation that was quite time consuming because of all permissions that need to be secured and a lot of processes for engaging at different levels,” said one Pact stakeholder. There were other elements that had to be juggled, including the challenge Pact had with its approach including integrating the sectors and initially setting out to cover a large geographic area. “Less focus on consortium partner direct activities, tighter integration in a smaller geographic location and more focus on strategic (local) partnerships would have enabled the project to do more sooner,” the stakeholder added. In general, the ET observed the level of frustration in ACE was higher in the field offices and at the implementation level. This includes staff members of the prime. “Personally, I am not very satisfied. I like the structure and


approach, but I am not satisfied with the outcomes because three years is a long time and there should be visual results,” said one Pact regional team member.

EQ4.B: FROM THE PERSPECTIVE OF THE CONSORTIUM PARTNERS

The three consortium partners brought considerable expertise and their existing networks of local partners to the ACE consortium. CPI, Mercy Corps, and SCI each had “strong relationships” and “clear lanes.” CPI had a long history working with EHOs. Mercy Corps had diverse in-country experience with agriculture, livelihoods, conflict sensitivity, and disaster risk reduction. SCI had a clear mandate and strong relationships with the Department of Education and government schools. The ACE structures is “not complicated but complex,” observed one partner. “It is hard to explain to an ordinary person with its many implementing partners, sectors, and geographic areas. It also makes it difficult to see the results of the project’s approach,” they observed. The collocation of staff in the regional offices and the higher level of communication and coordination this promotes is what partners cite as the most positive aspect of working in the consortium. “On the positive side, it reduces the need to rent office space,” said one staff member.

While each member of the consortium has its strengths and competitive advantages, the ACE consortium has lacked a strategy on how to engage and maximize them. “It is not an integrated approach, but a cooperative approach,” said one partner. While the original idea of an integrated approach with each partner contributing its expertise to the same community was a good one, they added, if the communities do not overlap it is difficult for the consortium partners to complement each other. When there were management challenges in the prime and other key consortium members, this impacted the effectiveness and speed of implementation in other parts of the project.

The project’s high turnover of senior management among the consortium partners in its first two years was “astonishing,” said one stakeholder. “Four organizations won this $50 million project and then people in high positions just disappeared so fast and this was disruptive for the whole consortium,” recalled one stakeholder. Project staff turnover slowed down planning, budgeting, and processing of grants. It eroded trust and confidence among local partners. The consortium structures require a lot of time to be spent on coordination, negotiating workplans, and submitting these up to USAID and feeding back comments down to the partners. Turnover of key personnel unnecessarily slowed these long lines of communications. Conversely, stability in consortium leadership in the last year since the arrival of the current Chief of Party (COP) and Deputy Chief of Party (DCOP) has seen sub-grants approved, internal communication streamlined, and its effectiveness increased.

This deep structural divide from the project’s beginnings and in the project’s management structure cannot be overcome by office collocation and field level staff getting along or working together in village-level activities. “At the very local level, you are not going to have a very junior team member integrating,” observed one consortium partner. “This is not the way the world works. It has to come from the higher level,” they said. At the implementation level, there appears to be a perception among some stakeholders that power is very centralized in ACE among the consortium’s senior leadership. “They say the consortium is really a combination of us, but the managers are not sitting here, they sit in Yangon,” observed a regional office staff member. “If we have a decision to be made, they only work by emails. In the field office, our consortium partners do their jobs, and we do our jobs.”

The consortium has some examples of teamwork, including internal integration with Pact, Mercy Corps, CPI, and SCO staff sharing space in two of the three regional ACE offices. Consortium members being collocated aids coordination, cooperation, and ongoing discussion about integration. “We are open-minded, can speak freely, and we feel like a team,” said one ACE staff member. “We can have proper discussions without email or phone calls,” they added. Prior to the offices being closed during pandemic, some found coordination to be management and meeting intensive. “We’re all very busy with our own


workloads and it is challenging for us to work on coordination with the other team,” observed another regional office staff member. “We have had many good experiences, but all the brainstorming is tiring,” they added. SCI co-locates with other consortium members in Hpa-An and Loikaw, but no longer in the Mawlamyaing office, after moving after Y1. When one partner is not present, electronic communication becomes the default and one stakeholder noted “we normally communicated through email.”

EQ4.C: FROM THE PERSPECTIVE OF THE LOCAL PARTNERS

Sub-grants to local partners are an important part of the ACE project. These grants to ESPs and CBOs have continued long-standing relationships of support that USAID has maintained with the communities of the southeast through cross-border support for many decades. These grants allowed the project to reach communities in areas controlled by EAOs that were inaccessible or off-limits to INGOs. They are welcomed by the grantees and the training provided by all ACE consortium partners that often goes with them is seen as adding important capacities to their own organizations. Training is provided by all Consortium partners, with CPI providing training to EHO partners, SCI providing child rights training to education and other partners, MC training partners on interest-based negotiation skills, and Pact providing a package of Organizational Development support tailored to each grantee. This vertical coordination and collaboration is important to note in this section on project structure. The initial round of CSGs focused on those ESPs providing health and education services. The RHELS grants that followed moved beyond service delivery to focus on promoting a rights-based approach as well as improving communication within these communities and strengthen their ability to conduct their own advocacy activities. More recently, the project has provided in-kind grants (IKGs), which in a list provided to the ET said the purpose of the awards were “to provide Personal Protective Equipment (PPE) and other materials” after the COVID-19 pandemic.34 The CSG grants were short-term and intended to fill gaps, with the safety and hygiene supplies grants intended to address pressing needs brought on by the crisis. The RHELS grants and the shift to a more political approach was something the Advisory Board had pushed recognizing the long-standing grievances held by these minority communities were behind the grievances and conflict in this region. While all useful, stakeholders said that looking back the project’s approach could have been less ad hoc and more strategic in using grants to develop partnerships.

The biggest complaint from local partners was the slow dispersal of sub-grants. ACE is a significant donor to the health and education systems in the EAO-controlled areas. While the three consortium partners all received their funding in September 2017 at the beginning of the project, the first eight sub-grants to local partners were made in Y1 between March and May 2018. VDFs received their first disbursements in August and September 2019, when 89 were made. Most grants were made in Y3, including 15 supporting ESPs, six to local partners supporting IBN, other disbursements to VDFs, and12 IKGs for PPE in September 2020. In Y3 ACE started 99 VDFs and funded 84 of them.

The Y2 RHELS grants were those that were subject to the longest delays. Stakeholders reported the reason to be unspecified internal management issues within the consortium prime, including the high-level turnover in the head office, including the departures of the country director, chief of party and deputy chief of party. As consortium staff changed, stakeholders said communication between ACE and potential grantees stopped, frustrating them. Local partners complained to USAID, but stakeholders said some of these grants took up to two and a half years to process. Even after USAID acknowledged the slower than expected transfer of funds, sub-grants to local partners were still not disbursed for another nine months following resolution of the issue holding up the sub-grants. Factors other than staff turnover might have

34 The materials supplied were mostly cloth masks and hand gel, which USAID does not define as personal protective equipment (PPE). These products were referred to as “safety and hygiene” supplies.


been responsible but for those now running the project were not in place at this time. Stakeholders within and without the project told how this “soured” relationships between ACE and its local partners.

Other issues include the balance in the project between direct implementation and sub-grants. Multiple stakeholders interviewed referred to the case of one former Advisory Board member who resigned earlier in the project, expressing disappointment that ACE didn’t do more to promote local organizations.35 After ACE increased its grant making to local organizations, the ET was told by a senior stakeholder that this ex-board member was pleased to see this increased support to local partners.

Grantees find USAID restrictions to be too harsh, including those on not funding small scale construction. “We are talking about areas where there is no clinic,” one stakeholder observed. “Either you build a health clinic or there is no clinic,” they added. This creates a bias against the most vulnerable communities and redirects resources to those villages that are better off at the expense of the neediest ones. Health providers find the project’s emphasis on training rather than providing medicine as misplaced. Grantees complain about the burden of administering grants and meeting all the consortium’s and USAID’s requirements.

EQ4.D: FROM THE PERSPECTIVE OF THE DIRECT BENEFICIARIES

During the evaluation design, the ET and stakeholders determined that direct beneficiaries would not be well-informed about the structure of the project they were interacting with. As was expected, enumerators found that when they called, respondents often had a low-level of awareness of ACE, the project. They did not identify as a participant in the ACE project, but rather as a member of their particular village-level committee, such as PTA, VHC, VDF, WASH, or WORTH group. For these reasons, such beneficiaries could not reflect on the structure and therefore the ET is unable to offer any findings on this part of EQ4.

CONCLUSIONS • From the perspective of the Prime, the project structure works well and allows it to draw on the

specialist skills and networks of the partners to achieve the overall objectives of the project. • From the perspective of the consortium partners, ACE’s centralized structure was seen as a weak

point when the prime experienced senior staff turnover and other internal management problems, resulting in slow processing of sub-grants.

• From the perspective of local partners, ACE’s consortium partners have done too much direct implementation at their expense, the project is bureaucratic, and sub-grants have taken too long process, approve, and disburse.

• From the perspective of the direct beneficiaries, their volunteer work is in the service of their own communities; they have a weak affiliation and do not identify as being part of the ACE project.

EVALUATION QUESTION 5 — PANDEMIC IMPACT

How has the COVID-19 pandemic; a). affected ACE’s ability to meet expected outcomes, intermediate results, and objectives, and b). resulted in the modifications to respond to the needs which arose out of the pandemic?

35 The person in question who resigned was not interviewed as part of this evaluation.


EQ5.A: AFFECTED ACE’S ABILITY TO MEET EXPECTED OUTCOMES, INTERMEDIATE RESULTS, AND OBJECTIVES

The COVID-19 pandemic and the subsequent response was an unforeseen phenomenon which has had a dramatic and unplanned impact on social, cultural, economic, and political activity in Burma. The ACE Y3 Annual Report stated that from the end of Q2 all the project’s activities have been affected by COVID-19 restrictions. While government restrictions in the capital and project area have not been uniform, “the trend across the board is toward significant delays in implementation.” ACE prioritized the safety of staff and beneficiaries, maintained close contact with local authorities to understand restrictions on implementation, and received USAID approval to provide IKGs of safety and hygiene materials to local partners. As face-to-face community engagement became impossible, the project moved to online and phone communication and turned to a mobile data collection tool for monitoring and evaluation.36

ACE has reported how it rapidly adapted its planned activities and responded in new ways. The project is acutely aware that it works with remote and marginalized communities with unreliable phone connections and limited internet connections cannot be moved online. Such adaptations have worked best with ACE local partners. During the brief respite before the second wave hit in August 2020, Pact was able to help villages set up bank accounts to allow VDF members to travel to the nearest township to pick up funds. MC negotiated with private sector partners to provide bulk agricultural inputs to villagers in addition to ongoing support they were receiving from the Department of Agriculture. USAID approved the reprogramming of funds towards COVID-19 preparedness and response, including hygiene kits for villages schools and PPE for EHOs.37

COVID-19 “upended” ACE’s whole model of community organizing that revolves around the community mobilization of communities. It undercuts the values of diversity and inclusion that the project seeks to implement, bringing people into process. Even with the adaptions, biases are introduced that are not yet fully understood. The requirement to hold small meetings of no more than five people will be exclusionary, noted one stakeholder, “and who will be left out? Will it be women?” The shift to support communities via phone or over internet apps favors the better off and less remote at the expense of the more marginalized and remote. “The internet connection is not very good in Kayin State, so online training is not very effective,” noted one stakeholder. After more than six months of travel restrictions, relationships with those communities that have no phone service is seen weakening.

The longer the pandemic continues, the more the adaptions by the project will have to move from quick fixes and improvisations to permanent changes in planned activities. Workplans that mention meetings and trainings are just being revised every three months and activities are pushed from one quarter to the next, one management-level stakeholder said. “But we don’t know when these activities can resume,” they observed. USAID is not yet suggesting changes to the activities and approving workplan postponements. “When COVID is over, we will have to speed up our activities. We may have to redesign our workplan,” said one implementer working on agriculture. But crops are dependent on seasonal inputs, including rain, and activities designed to support farmers cannot be easily rescheduled. “If we miss a season, then USAID will need to approve a no-cost extension for us,” said another agricultural implementer.

Consortium partners at all levels have been frustrated by the restrictions and slow decision-making by USAID during the pandemic. The Agency took months to approve reprogramming for COVID-19 related activities, including allowing the supply PPE to partners, although these restrictions often originated from a high-level in Washington. D.C. “We need to develop a strategy that acknowledges the reality of the

36 “Y3 Annual Report”, op. cit., p.1. 37 Ibid. p.32.


situation we’re dealing and how long this is going to be present in Myanmar,” said one consortium partner. “We have been really restricted in thinking about adaptations and what we can do because of USAID’s very bureaucratic approach for the first six months,” they said. With less than two years to go, ACE consortium members at all level are aware that time is running out on the project. “If the restrictions last longer than expected, we may have to change our activities. We are already in Y4 and we have only one more year,” one said. Other international organizations in Myanmar are already planning for COVID-19 restrictions to continue, in some form, until 2022, asking all implementers to produce COVID risk assessment plans. Some are starting to reflect on what they have been able to achieve despite the restrictions and the role local partners have played in ensuring the continuation of activities. “You can call it localization, but we are trying to acknowledge and appreciate the efforts of local organizations that have not only managed the COVID response but are key to all our programs,” one international stakeholder said. Local partners have managed to get on without INGOs being able to travel to project sites. “What does that mean for the international partners?” they asked.

EQ5.B: RESULTED IN THE MODIFICATIONS TO RESPOND TO THE NEEDS WHICH AROSE OUT OF THE PANDEMIC

Stakeholders tell a familiar story of events quickly cancelled or postponed. “Everything came to a screeching halt,” one stakeholder recalled. But not all the project components were impacted in the same way. The application of restrictions on travel and public meetings by government and EAOs were also not initially uniform in their initial application in April, then relaxation in August, and finally reimposition in September. It took the project and its partners time to fully understand the situation.

Organizations that for years had worked across the Myanmar-Thailand border found that trainers and participants were unable to travel across the frontier once movement restrictions were put in place. Agriculture trainings that relied on demonstrating techniques were not possible when staff were stopped from traveling. Some farmers can be coached over the phone, but not all of them have a connection or feel comfortable speaking over the phone. With these activities often timed to match the season, they were most sensitive to any extended delays. Health clinics remained open for primary care but making secondary referrals to township level centers became more difficult. Local partners have been trained over the internet where possible. Consortium staff have used the time to develop new written materials, translations, and guidelines in local languages. Schools were on their annual summer holidays in April when the pandemic restrictions first hit and with ACE’s support on a natural hiatus the negative impact was delayed until they were unable to return in June. Home-based learning kits have now been prepared to help parents instruct students at home. Organizers are coaching parents over the phone and producing printed materials for distribution to those without access to such devices. Online curriculums and trainings have been developed for teachers. While government schools have remained closed, some ESP schools have remained open.

Stakeholders working at the implementation level said they had been told by volunteers that some of their groups were still regularly convening at the village level. Respondents to the mini-survey reported lower levels of activity since March, with an average of 40 percent across of all types of groups never having met since the pandemic began (Figure 8).


Figure 8. Frequency of Village Meetings Before and After COVID

There was significant variation observed between activity level among the types of groups. The mini-survey data shows that these groups of direct beneficiaries have reacted differently to the pandemic restrictions. Of those surveyed, PTAs were most inactive with 58 percent never having met since March. Of the Health and WASH committees were also above average in their inactivity, 54 percent had not met at all in the last eight months. The governance engagement groups, and WORTH groups showed the highest level of activity, with only 32 percent and15 percent respectively reporting their organizations had not met since March.

If groups stop meeting, they stop organizing and working together to improve their communities. “We know that some of the work of the committees is ongoing, it is still happening, we just have a limited ability to take it to the next level,” said one consortium member. Support for these bodies is iterative and at many distinct stages. One training builds on the next in a cycle that lasts about 18 months, the project staff member described. When contact does restart, trust will have to be rebuilt with communities and reengaging as the virus has created new fears and anxieties about engaging with outside groups. “We can assume that this is going to be really challenging,” said one stakeholder.

CONCLUSIONS • The ACE consortium has adapted to the current pandemic restrictions by maintaining relationships

with the partners and communities, but the project’s objectives and outcomes are unlikely to be met if pandemic restrictions continue.

• There is growing recognition that the pandemic restrictions may well go on for some years and that this may require ACE to make significant changes to its design and activities.

EVALUATION QUESTION 6 — RECOMMENDATIONS

Where are the potential opportunities to alter, scale-up, reinforce, or cancel activities or approaches over the remainder of the program in response to the changing context and political dynamics in the implementation areas?

This section provides recommendations that could be implemented either in the near term, before the project ends, or in a future phase. Based on guidance from USAID, the ET assumes no extension to the


current period of performance, which means that there may be only 18 months of programming before ACE must begin closing down.

ALTER

As described above in EQ5, the ACE activity’s community organizing model has been challenged by the pandemic and the consortium has already had to adapt. While ACE did change IRs in its Objective III in Y4 with USAID approval to better reflect long standing challenges to working with government, it has not yet changed its outcomes, objectives, or core activities as a result of the pandemic. ACE continues to work with the same key partners and sets of direct beneficiaries while it has adapted the way it interacts during the pandemic restrictions. Where possible, communities have been contacted by phone or internet app rather than in person meetings. Stakeholders say this mode of communication has worked best for sub-grantees whose organizations are based in larger towns and that can afford reliable internet access. But such types of contact are often not possible, especially when the community member is remote or does not have phone service. In these cases, the relationship has been interrupted.

Stakeholders said some government partners have been too busy to interact, even online, during the pandemic. Given senior management of ACE is aware that other international organizations in Myanmar are reportedly planning for the impact of the pandemic to last until 2022, it was striking that most implementation level ACE staff saw such remote interactions as a temporary measure. There seemed to be a mismatch between high-level and working-level thinking. Pandemic restrictions on movement and meetings were suddenly imposed in March and lifted for a brief time in August before being reimposed in September. Staff naturally express the hope that if the project can “return to normal,” then they will be able to pick up where they left off in their workplans. However, the longer the pandemic restrictions are in place, the greater the potential for inertia to set in, for knowledge to be lost among beneficiaries, for government counterparts to rotate, and for restarting the project to become more difficult and time consuming. In revising timelines, the implications of these extended delays will have to be considered by USAID and the project.

Some ACE workstreams are better-suited than others to adapt to the crisis posed by the pandemic. Health and WASH activities and messages that promote good hygiene are as important as ever and can improve community resilience to COVID-19 and other diseases. USAID has approved the reprogramming of funds from activities not completed and allowed them to be used for purchase of hygiene kits for communities, and the provision of cleaning supplies for schools. The IKGs for safety and hygiene supplies, made in September 2020, were valued at about $461,000 (MMK 645,531,500). They were not reprogramming, according to USAID guidance, but USAID-approved subawards issued in response to COVID.

The use of IKGs to address urgent pandemic-related needs is something of a conceptual shift for the project. By making these grants, ACE is taking a step away from being about changing the way community governance to responding to immediate community needs. Given USAIDs long-term humanitarian support in this region and support to multi-donor trust funds such as MHF and LIFT, it is familiar role for the agency. But if the restrictions on project activities continue, the IKGs provide an example of how to alter project programming in response to the current context.

Other alterations that might require distinct kinds of support have been discussed and might require more planning. One EHO has expressed interest in building its capacity to do COVID-19 testing. This is a complex alteration as enhancing the EHO’s ability in laboratory testing may require outside technical assistance beyond the ACE project’s current capacity.


Recommendations • ACE should assess the impact of COVID-19 restrictions on all the village level organizations it works

with to better understand apparent variations in resilience during the pandemic and identify the groups which are most able to continue working.

• If ACE activities cannot resume as normal and remote or small group interactions become the norm for the rest of the project, which is increasingly likely, then expectation of project results need to be revised downward.

• Given the high-level of uncertainty produced by the pandemic, which is expected to persist for some time, building in flexibility and adaptability into any future alterations will be important.

• Opportunities to reprogram unused funds from activities to support partners with in-kind donations should continue to be actively explored.

• While the ET assumes no extension to the project, if this were be to be changed by USAID, then it offers an opportunity to maintain the original set of activities and results on an adjusted or extended timeline.

SCALE UP

ACE has been scaling up to new geographic areas, with new VDFs being formed and funded as recently as late in Y3. ACE is now working in 29 townships, adding 362 new villages in Y3, which represents a 25percent increase in coverage in that project year. The Y3 Annual Report noted that Pact has now reached its target number of villagers. Stakeholders with this perspective said ACE should be consolidating its efforts and looking for ways to sustain gains already made. Establishing a new village-level activity can take over a year. USAID has signaled to ACE that it should work within its current project period. With the incorporation of a project wind-down period, there will soon be insufficient time for the project to start any new community organizing activity.

Several stakeholders saw the idea of scaling up as meaning that the project could focus or better target its resources, especially through increasing grants to local partners. “If we don’t have a vaccine in Myanmar, then we will see restrictions for at least another half a year. It is a good mitigation measure to say, ‘let’s channel more funding through local partners who can work while we can’t,’” said one stakeholder. However, other stakeholders noted that this may be an option for some but is not a solution for all. Local partners are not immune from the same travel and meeting restrictions delaying direct implementation by the project. Developing a nuanced understanding of which partners are continuing to provide services under these trying circumstances and providing them with additional support to continue their activities could Others suggested that ACE could boost its financial support to VDFs and WORTH groups. Every VDF supported by ACE receives a one-time micro-grant to boost the fund. The project had recently more than tripled funding of VDFs from $300 to $1000 and this higher level of support could be maintained. ACE is already planning to issue more than 200 VDF micro-grants, which would be one for each direct implementation village.

Recommendations • USAID and the ACE project should continue to explore the possibility of using existing mechanisms,

channels, and community relationships to increase funding to local partners, including through cash grants, providing additional micro-grants to VDF, as much as their capacity allows.

• All residual community organizing activities should be reviewed and assessed on a rolling basis to understand the point at which they may no longer be feasible to be completed or adapted during ACE’s current project period.


REINFORCE

Rather than scale up, many stakeholders believe the consortium should be reinforcing its local partners. These stakeholders believed that ACE should explore other mechanisms to use unspent funds to support partner communities using extra honoraria or cash transfers. Other stakeholders mentioned that existing grants mechanism, such as RHELS, could be used to expand already planned partnerships with local organizations by increasing the scope of the disaster risk management activities or moving into new areas such as land reform or supporting local government reforms being implemented by the General Administration Department. Given the COVID-19 restrictions, stakeholders believe that reinforcing existing relationships and funding arrangements would be better than starting on new ones.

ACE has a learning agenda and a set of learning questions as part of its approved MEL plan, along with a number of planned research activities, such as the APEA Updates, Participatory Action Studies, and WORTH Assessment.38 The ET has in the course of its work identified a number of observations that deserve further study, including from data gathered by the mini-survey. For example, and as noted above, the WORTH groups that are mostly women have met more often during the pandemic than WASH groups dominated by men. What explains this apparent greater resilience? The mini-survey is a sample, but what would be revealed if the project was to do a more detailed audit of village level group activities during COVID? Further in-depth study the relative strengths and weaknesses of the Integrated Service Model or Community Leadership Model might be appropriate.

Recommendations • Given the ability of many local partners to continue to function and serve their communities during

the pandemic ACE should explore how existing sub-grant mechanisms can be reinforced or added to support their work, including the possibility of cash grants.

• ACE could study those village level groups and local partners that have been able to remain active during the pandemic to learn from, document, and share this experience with others in the project.

CANCEL

In discussions about the challenges of implementing the ACE community organizing model during the pandemic, some stakeholders acknowledged reviving stalled parts of the project may be challenging if the pandemic restrictions continue for much longer. “Anything that is being done by consortium partners that hasn’t started yet is going to struggle to get going because almost all of our direct activities are reliant upon large scale community engagement,” observed one stakeholder. This model relies on bringing together sizeable groups of people and pandemic-imposed meeting restrictions mean that these gatherings will not be possible. With many stakeholders expecting pandemic restrictions to continue for months, if not years, it may be some time before unrestricted meetings are allowed and the communities feel comfortable with them. “Those types of activities are going to have a hard time starting up in Y4,” said the stakeholder. As noted by several stakeholders, the project may have only 18 months of effective programming time before it will need to begin closing. Stakeholders said there exists opportunities for adaption ahead of a general lifting of restrictions. The mosaic of governance can work in ACE’s favor. The project has observed significant variation across the region in terms of types of activities and sizes of groups that different authorities will permit. Whether an area is controlled by the government or EAOs, ACE follows whatever guidance is provided and ensures village authorities are on board with its approach before entering communities.

38 See, for example, “Table 5: Initial ACE Learning Questions”, ACE MEL Plan, September 1, 2020, p.20-24.


While the sense of urgency among stakeholders is understandable, rushing to restart community activities with strict controls on meeting size could have unintended consequences for women’s participation and equity. ACE’s community-led initiatives are meant to be inclusive and representative. If they are not, this would show short-term activity but in the long run undermine the project’s Objective II of strengthening community participation.

All stakeholders have noted that building community trust and relationships in the southeast takes time. DRR activities were planned with extensive community assessments and consultation. Such confidence building may not be possible either before the next flood season or when the project ends. “But if we are forced to stop halfway through the process, all the effort will be lost,” said one stakeholder. Working with famers is also time sensitive. Agricultural assistance must be timed to the seasons and may not be amenable to quickly restarting. In all these activities, the pandemic presents a new risk that needed to be assessed. Restarting only to have to stop again would undermine confidence of the staff and trust of the communities, stakeholders said. It could cause more harm to the project, than good to the intended beneficiaries.

Recommendations • With the project having less than two years remaining, a decision to extend or not will need to be

made soon. • USAID and ACE should begin planning responses to varying pandemic scenarios in Myanmar,

determining when each activity component would need to phase out to ensure successful completion and closeout.

• ACE should consider prioritizing interventions that are more resilient to periodic shutdowns for the anticipated life of the project.

• ACE should assess the implications for women’s participation and inclusivity related to partially restarting project activities.

• ACE must realistically assess how long any new grants would take to complete, ensure they can be completed in the minimum remaining project period, and commit to those timelines.

• With this information, USAID/Burma and ACE should consider identifying “points of no return” for each project component after which they must be phased down and/or cancelled.


ANNEXES

ANNEX I: EVALUATION STATEMENT OF WORK

1. TITLE OF ACTIVITY

Mid-term Evaluations of the Advancing Community Empowerment (ACE)

Estimated period of performance October 2020 through April 2021

2. OBJECTIVES

The purpose of the evaluation is to assess the progress of ACE towards achieving intended outcomes, intermediate results, objectives, and overall goals relative to the current political context and impacts of COVID-19; identify best practices and lessons learned related to health, education, and multi-sectoral/integrated approaches to improving governance; and provide recommendations for the remaining two years of the ACE activity and future programming.

3. AUDIENCE AND INTENDED USE

The primary audience for this evaluation will be USAID/Burma and specifically the Office of Democracy, Governance, and Humanitarian Assistance, Office of Public Health, and Program Development Office. Relevant components and information from the evaluation will be shared with implementing partners, sub-partners, and the broader development community at the Mission’s discretion.

4. BACKGROUND

Description of Activity to be evaluated (ACE)

Activity/Project Name Advancing Community Empowerment (ACE) 

in Southeastern Myanmar

Implementer Pact, Inc.

Cooperative Agreement/Contract # 

AID-482-A-17-00001

Type of Evaluation Mid Term Performance Evaluation

Total Estimated Ceiling of the Evaluated Project/Activity(TEC) 

$48,000,000 

Life of Project/Activity  August 2017 to August 2022


Active Geographic Regions Demoso, Hpruso, Hpasawng (Kayah State)

Pekon (southern Shan State)

Hlaingbwe, Kyainseikgyi and Thandaunggyi (Kayin State) and Thanbyuzayat (Mon State)

In addition to the above geographic regions for direct implementation, the ACE activity also supports sub-grants to local organizations in:

Pyinmana (Nay Pyi Taw)

Kyaukkyi, Shwegyin, and Taungoo (Eastern Bago Region)

Bilin, Mudon, and Ye (Mon State)

Dawei, Tanintharyi, and Yebyu (Tanintharyi Region)

Hsihseng and Pinlaung (Southern Shan State)

Bawlakhe, Loikaw, Mese, and Shadaw (Kayah State)

Hpa-An, Hpapun, and Myawaddy (Kayin State)

 

Development Objective(s) The activity’s goal directly contributes to the USAID mission objective of  “Strengthen institutions to promote good governance, rule of law and human rights, and to expand space for civil society and the media to support the democratic transition ” and USAID Mission’s Democracy, Governance and Humanitarian Assistance (DGHA) Office strategy “Democratic Governance Strengthened to Advance the Transition ”. 

USAID Office USAID Burma, Office of Democracy, Governance, and Humanitarian Assistance

Required Evaluation The evaluation meets the requirement under ADS 201.3.5.13

External or internal evaluation External

4.1 EVALUATION QUESTIONS

The TO Contracting Officer’s Representative (COR) at USAID’s Regional Development Mission for Asia (RDMA), and the Activity Manager and DGHA team at USAID/Burma will work with the contractor to refine and clarify the evaluation questions.

ACE Mid-term Evaluation

1. How have externalities related to the evolving political context in ACE implementation areas impacted ACE’s ability to achieve expected objectives over the first three years of the project?

2. To what extent have ACE’s health, education, and gender-based violence (GBV) interventions contributed to achieving it’s expected outcomes, intermediate results, and objectives at this point in the project implementation? Specifically, in the following areas:

a) Health care access and coordinated service delivery improved


b) Education access and services improved

c) GBV awareness increased

3. To what extent has a multi-sectoral and integrated community development approach contributed to achieving expected outcomes, intermediate results, and objectives in the areas of strengthening community participation and local governance?

4. How has the organizational structure of the ACE activity (a combination of direct implementation, consortium partners, and sub-partner grants) affected ACE’s ability to achieve desired outcomes and results at this point in the project implementation?

a) From the perspective of the prime implementing partner

b) From the perspective of the consortium partners

c) From the perspective of the local partners

d) From the perspective of the beneficiaries

5. Where are the potential opportunities to alter, scale-up, reinforce, or cancel activities or approaches over the remainder of the program in response to the changing political dynamics in the implementation areas?

6. How has the COVID-19 pandemic:

a) affected ACE’s ability to meet expected outcomes, intermediate results, and objectives?

b) Resulted in the modifications to respond to the needs which arose out of the pandemic?

4.2 EVALUATION DESIGN AND METHODOLOGY

For the ACE evaluation, it is anticipated to use a mixed methodology with data sources, including but not limited to ACE work plans, progress reports, researches, assessments, surveys, studies, evaluation reports, and M&E data and reports etc. Below are the illustrative evaluation designs and data collection methods for both evaluations. But the contractor is encouraged to propose their own design matrix referencing/expanding these illustrative design and methods.

Questions Suggested Data Sources

Suggested Data Collection Methods

Suggested Data Analysis Methods

EQ1. How have externalities related to the evolving political context in ACE implementation areas impacted ACE’s ability to achieve expected objectives over the first three years of the project?

Remote interviews

ACE progress reports

• Qualitative (key informant interview and/or focus group discussions etc. as relevant).

• Desk review, secondary analysis as necessary.

To be proposed by the contractor

 





EQ2. How have ACE interventions contributed to achieving it’s expected outcomes, intermediate results, and objectives at this point in the project implementation?

ACE progress reports, work plans, M&E Data, Baseline report

• ACE’s Quantitative M&E data analyses.

• Desk review, secondary analyses as necessary.

• Qualitative: Phone interviews.

• Household surveys (if feasible) or Remote surveys among the beneficiaries of ACE using mobile data collection platform like COMM Care.


 

EQ3. How has the organizational structure of the ACE activity (a combination of direct implementation and sub-partner grants) affected ACE’s ability to achieve desired outcomes and results at this point in the project implementation?

Remote interviews





 

EQ4. To what extent has a multi-sectoral and integrated community development approach contributed to achieving expected outcomes, intermediate results, and objectives in the areas of strengthening community participation and local governance?

ACE progress reports, work plans, M&E Data

• ACE’s Quantitative M&E data analyses.

• Desk review, secondary analyses as necessary.

• Qualitative: Phone interviews.


 

EQ5. Where are the potential opportunities to scale-up or reinforce activities or approaches over the remainder of the program in response to the changing political dynamics in the implementation areas?

Remote interviews





 





EQ6. How has the COVID-19 pandemic:

a) Changed ACE’s ability to meet expected outcomes, intermediate results, and objectives?

b)Resulted in the modifications to respond to the needs which arose out of the pandemic?

Remote interviews




 

4.3 DELIVERABLES AND REPORTING REQUIREMENTS

The contractor must submit all the deliverables listed below. Specific timeline for each deliverable for each evaluation is to be discussed later once the award is made and after the kick-off call. The Mission tentatively plans that the ACE evaluation will take place on/about October 2020.

1. Evaluation Design and Work Plans.

Based on desk review findings, the evaluation teams must submit an evaluation design and work plan to Task Order Contracting Officer Representative (TO-COR) and Activity Manager no later than 10 business days after the kick-off call. The evaluation designs must include:

● The detailed evaluation design matrix that link the final evaluation questions to data sources, methods, and the data analysis plan;

● Both quantitative and qualitative data collection instruments or their main features, with sex-disaggregation of all people-level data;

● A list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); and

● Known limitations to the evaluation designs. The work plan shall include: (1) activities and anticipated schedule and logistical arrangements; and (2) list of evaluation team members, delineated by roles and responsibilities.

The TO-COR, Activity Manager, and USAID/Burma DGHA team will review and provide feedback to the draft evaluation design and work plan within 10 business days.

The evaluation teams will finalize the evaluation design and work plan within 5 business days after the in-brief with USAID team. The work plan must also specify what support will be needed from the Mission in terms of securing travel authorizations and meetings with government officials and counterparts.

2. In-briefings:

The contractor’s evaluation team shall organize in-person or virtual (if the COVID-19 slows down by that time) in-brief meetings with the TO-COR, USAID/Burma’s Program Development Office


(PDO), and the DGHA. During these briefings, the evaluation teams will present their plan for field work, discuss the methodology, review the work plan and make any additional adjustments as required.

3. Exit Briefings:

The evaluation teams shall hold in-person or virtual exit briefings prior to leaving the country, including a presentation of the draft evaluation findings to the USAID/Burma Mission, to discuss the status of data collection and preliminary findings.

4. Draft Evaluation Reports:

The draft evaluation report must be submitted to the TO-COR, and Activity Manager within 20 business days after the exit briefing and comply with the requirements described in Section C.6.

5. Final Evaluation Reports:

Within 7 business days of receipt of USAID’s comments to the draft evaluation reports, the contractor shall submit a final evaluation report, which responds to/incorporates the final comments from TO-COR approval and Activity Manager’s clearance.

5. EVALUATION TEAM COMPOSITION AND QUALIFICATIONS

Key Personnel:

One (1) evaluation team lead/integrated community development expert with experience evaluating multi-sectoral, integrated community development approaches using mixed methods. Experience in Asia and/or Southeast Asia required and experience in Burma highly desired. The Team Lead will have ultimate responsibility for the technical approach, analysis, findings, recommendations and successful management of the evaluation.

One (1) research specialist with experience evaluating humanitarian or community development approaches and extensive knowledge of the operating environment in Burma. Must speak, read and write Myanmar language and English. S/he will be responsible for ensuring technically sound information and analysis throughout the planning, data collection, analysis and reporting processes. S/he will ensure that the data gathered adequately addresses the evaluation questions.

All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest.

The evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy (Attachment 1) and guidance included in the USAID Automated Directive System (ADS) in Chapter 201.

6. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT

The Contractor must provide a complete and separate evaluation report for each of the evaluations conducted.

Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, the draft and final reports will be evaluated against the following criteria to ensure quality.

● The report must represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the project.

● The evaluation abstract of no more than 250 words should describe what was evaluated, evaluation questions, methods, and key findings or conclusions.

https://www.usaid.gov/sites/default/files/documents/1870/201maa.pdf

https://www.usaid.gov/sites/default/files/documents/1870/201maa.pdf


● The Executive Summary of the report must present a concise and accurate statement of the most critical elements of the reports.

● The report must adequately address all questions included in the SOW, or the questions subsequently revised and documented in consultation and agreement with USAID.

● The evaluation methodology must be explained in detail and sources of information properly identified.

● Limitations to the evaluation must be adequately disclosed in the reports, with particular attention to the limitations associated with the methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).

● Findings must be presented as analyzed facts, evidence, and data (with sex, age and ethnic disaggregation, as appropriate) and not based on anecdotes, hearsay, or simply the compilation of people’s opinions.

● If findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific.

● Findings and conclusions must be specific, concise, and supported by strong quantitative or qualitative evidence.

● Further resources and guidance on USAID Evaluation policy, preparing the evaluation report and report templates could be found at:

o USAID Evaluation policy

o How-To-Note: Preparing evaluation report

o Evaluation Report Template

7. OTHER REQUIREMENTS

All modifications to the required elements of this Service Request, whether in technical requirements, evaluation questions, evaluation team compositions, methodology, or timeline, need to be agreed upon in writing by the TO-COR. Any revisions must be updated in the Service Request and only the final version shall be included as an annex to the Report.

LIST OF ANNEXES (TO BE SHARED SEPARATELY)

A. Background, problem and context of ACE

B. ACE Theories of Change and Results frameworks

C. ACE Activity MEL plans

D. ACE Work plans and progress reports

https://www.usaid.gov/sites/default/files/documents/1870/USAIDEvaluationPolicy.pdf

https://usaidlearninglab.org/library/evaluation-report-template


ANNEX II: SOURCES OF INFORMATION

DOCUMENTS AND SOURCES

ACE Consortium Y1 First Quarterly Report (Aug–Sep 17)

October 2017

ACE Consortium ACESM Conflict Sensitivity and Mitigation (CSM) Framework

November 2017

ACE Consortium Y1 Q1 Report (Oct–Dec 17) + annexes

January 2018

ACE Consortium Advancing Community Empowerment in Southeastern Myanmar Year 1 Workplan

April 2018

ACE Consortium Y1 Q2 Report (Jan–Mar 18) + annexes

April 2018

ACE Consortium ACE PEA Final July 2018

ACE Consortium Y1 Q3 Report (Apr–Jun 18) + annexes

July 2018

ACE Consortium Y1 Annual Report (Oct 17–Sep 18) + annexes

November 2018

Myanmar Information Management Unit (MIMU)

Main Spoken Languages of Myanmar January 2019

ACE Consortium Advancing Community Empowerment in Southeastern Myanmar Year 2 Workplan

January 2019


January 2019


April 2019

ACE Consortium (Mercy Corps)

ACE Governance Analysis July 2019


July 2019

ACE Consortium Y2 Annual Report (Oct 18–Sep 19) + annexes

November 2019


January 2020



April 2020

ACE Consortium ACE EICS Strategy June 2020


July 2020

International Crisis Group Identity Crisis: Ethnicity and Conflict in Myanmar

August 2020

ACE Consortium Y4 MEL Plan 1 September 2020

ACE Consortium Youth Engagement Approach September 2020

ACE Consortium Advisory Board Action Tracker 27 September 2020

ACE Consortium Staff contact list 6 October 2020

ACE Consortium Volunteer Committee contact list 7 October 2020

ACE Consortium Success Stories Highlights 8 October 2020


LIST OF KEY INFORMANT INTERVIEWS

# Organization Contact Person or Interviewee

Location/Interview Language

Youth Sex

1 USAID, DGHA Senior Management Yangon/English N M 2 USAID, DRG Senior Management DC/English N F 3 USAID/South Sudan Senior Management Juba/English N F 4 USAID/Burma Current AOR Yangon/English N M 5 UNOCHA Myanmar Humanitarian Fund

(MHF) Yangon/English N M

6 UNOPS Livelihoods and Food Security Fund (LIFT)

Yangon/English N F

7 ACE Advisory Board Independent Consultant Yangon/English N M 8 ACE Advisory Board Independent Consultant Yangon/English N F 9 Pact HQ Staff Yangon/English N M 10 Pact HQ Staff Yangon/English N F 11 Pact HQ Staff Yangon/Myanmar N F 12 Pact Field Staff Hpa An/Myanmar N M 13 Pact Field Staff Hpa An/Myanmar N F 14 Pact Field Staff Mawlamyaing/Myanmar N F 15 Pact Field Staff Mawlamyaing/Myanmar N M 16 Pact Field Staff Mawlamyaing/Myanmar N M 17 Pact Field Staff Loikaw/Myanmar N F 18 Pact Field Staff Loikaw/Myanmar N F 19 Pact Field Staff Loikaw/Myanmar N M 20 Pact Field Staff Hpa An/Myanmar N M 21 Pact Field Staff Hpa An/Myanmar N F 22 Pact Field Staff Hpa An/Myanmar N M 23 Save the Children HQ Staff Yangon/English N M 24 Save the Children Field Staff Hpa An/Myanmar N F 25 Save the Children HQ Staff Hpa

An/Yangon/Myanmar N F

26 Save the Children Field Staff Loikaw/Myanmar N M 27 Community Partners

International HQ Staff Yangon/English N M

28 Community Partners International

HQ Staff Yangon/Myanmar N M

29 Community Partners International

HQ Staff Loikaw/Myanmar N M

30 Mercy Corps HQ Staff Yangon/English N M 31 Mercy Corps Field Staff Hpa An/Myanmar N M 32 Mercy Corps Field Staff Hpa An/Myanmar N F 33 Mercy Corps Field Staff Hpa An/Myanmar N M 34 Mercy Corps Field Staff Loikaw/Myanmar N F 35 Grantees (IBN) Field Staff Hpa An/Myanmar N F 36 Grantees (IBN) Field Staff Mawlamyine/Myanmar N M


# Organization Contact Person or Interviewee

Location/Interview Language

Youth Sex

37 Grantees (CSG/RHELS)

Field Staff Kayar State/Myanmar N F


HQ Staff Yangon/Myanmar N M


Field Staff Mae Sot, Thailand/Myanmar

N M


Field Staff Mae Sot, Thailand/Myanmar

N F


Field Staff Mon State/Myanmar N F


Field Staff Thailand/Myanmar N F


Field Staff Kayah/Myanmar N M


ANNEX III: DATA COLLECTION INSTRUMENTS

USAID BURMA - ACE EVALUATION

KEY INFORMANT INTERVIEW DONORS, IMPLEMENTERS & ADVISORY BOARD

Researcher Name:  

Notetaking method: [Recorded and transcript/computer notes/handwritten notes/notetaker]

Interview Date:

Respondent Name:  

Respondent Position:  

Respondent Sex:

Respondent Ethnicity [if Burmese]:

What is your ethnicity? [Ask Burma respondent to self-identify the ethnicity]

Respondent age: Are you between 18-29 years old? [If under 18, terminate the interview. If YES, classify as youth]

Respondent Region:

Start Time:  

End Time:  

Hello, I am _______________ and I work with Social Impact, an independent company conducting research for the United States Agency for International Development (USAID) Burma Office. We are conducting an evaluation of the Advancing Community Empowerment in Southeastern Myanmar (ACE) project. We intend to interview approximately 40 key stakeholders in this project, and you have been selected to participate because we believe you can provide a unique perspective on the work of ACE. Specifically, we are trying to learn whether ACE has contributed to improvements in areas such as community capacity, health, education, and livelihoods, and how well the program as adapted to contextual challenges such as politics and the COVID-19 pandemic.

Your participation in this interview should not pose any harm or discomfort to you. Indeed, your participation in this interview is completely voluntary and you are free to decline to respond to any question for any reason and to end the interview at any time with no consequence. You will not be provided with any payment for your participation in this interview. Your honest responses will help USAID understand the results of ACE.

Your personally identifiable information, such as your name and phone number, will not be connected to any information you provide in this interview. The data from this interview will be used by the Social Impact team for analysis purposes only and your answers will only be reported as part of an aggregated data set. This report will be shared with ACE and made publicly available on USAID DEC. To aid our notetaking, we would like to record the interview, but you may choose to participate and not be recorded, with no consequence. If we record this interview, the recordings uploaded to a password protected site and will not be made public.


This interview will last approximately one hour. Do you understand that your participation in this interview is voluntary, unpaid, and that you can end the interview at any point? If you have any follow up questions, please contact Myint Zaw at 09450064401 or Christine Thomas at [email protected]. You may also reach the Social Impact IRB in the U.S. at [email protected] or +1 703-465-1884.

Are you willing to participate? Yes___ (check) No___ (check)

Do you allow voice recording of the interview? Yes___ (check) No___ (check)

# Questions Guidance, Adaptions, & Prompts

1 Can you describe your role within your relationships to and level of involvement with the ACE project?

Prompt: How long have you been in this position and what ways have you been directly or indirectly interacted with ACE? Prompt: What responsibilities do you have with regards to these interactions with ACE? Prompt: What activities are you directly involved with regards to ACE? Prompt: Did you have any role in the design of this project?

EQ 1: How have externalities related to the evolving political context in ACE implementation areas impacted ACE’s ability to achieve expected objectives over the first three years of the project? 2 At the time ACE begun, how appropriate was the project

design to the political context? Note: If the interviewee was not involved in the design or beginning some of these questions could be skipped Prompt: Or in other words, how favorable was the political context to the project as designed when it started?

3 How has the political context of southeastern Burma changed during the life of the project (since August 2017)? Or didn’t change? Why hasn’t the anticipated repatriation of refugees from Thailand happened?

Prompt: Describe and give examples of these changes?

4 How have these changes impacted your organization’s operations and activities? How do you understand these changes have impacted ACE activities?

Note: Separate prompts depending on role/activity of donor or implementer Prompt: How have the changes impacted health care access and coordinated service delivery? (IR 1.1)

mailto:[email protected]



# Questions Guidance, Adaptions, & Prompts Prompt: How have the changes impacted improving education access and services? (IR 1.2) Prompt: How have the changes impacted improving sustainable livelihoods? (IR 1.3) Prompt: How have the changes impacted improving WASH? (IR 1.4) Prompt: How have the changes impacted increasing community capacity for preparedness, mitigation, and responses to disaster? (IR 1.5)

5 How have these impacted the involvement of the local community?

Note: Separate prompts depending on role/activity of partners or grantee Prompt: How have the changes impacted improving community capacity and leadership skills? (IR 2.1) Prompt: How have the changes impacted improving participatory community decisions making? (IR 2.2) Prompt: How have the changes impacted developing a community action plan? (IR 2.3) Prompt: How have the changes impacted enhancing trust and reconciliation? (IR 2.4) (EQ3) Prompt: How have the changes impacted increasing youth and/or women’s participation? (IR 2.5) (EQ3)

6 How have these changes impacted the involvement of local government?

Note: Separate prompts depending on role/activity of partners or grantee Prompt: How have the changes impacted increasing the integration of public service provision? (IR 3.1) (EQ3)


# Questions Guidance, Adaptions, & Prompts Prompt: How have the changes impacted strengthening local public service provision? (IR 3.2) (EQ3) Prompt: How have the changes impacted increasing trust and collaboration between communities and local authorities? (IR 3.3) (EQ3) Prompt : What is the impact of the turnover of govt officials in key positions at the township level on program implementation? Prompt : Which change has a greater impact on the ground? What are some examples of where changes in local govt personnel have had negative or positive impacts on project implementation?

EQ2: To what extent have ACE’s health, education, livelihoods, and DRR interventions contributed to achieving its expected outcomes, intermediate results, and objectives at this point in the project implementation? Specifically, in the following areas:

a) Health care access and coordinated service delivery improved b) Education access and services improved c) Sustainable livelihoods promoted d) WASH improved e) Community capacity for preparedness, mitigation and responses to

disaster increased 7 How effective has ACE been in achieving its goal of

reducing community vulnerabilities in its key focus areas of health care access, education, sustainable livelihood, WASH, and disaster preparedness?

Note: Separate prompts depending on role/activity of partners or grantee

How effective (or non-effective) has ACE been in improving health care access and coordinated service delivery? (IR 1.1) How effective (or non-effective) has ACE been in improving education access and services? (IR 1.2)


# Questions Guidance, Adaptions, & Prompts How effective (or non-effective) has ACE been in promoting sustainable livelihoods? (IR 1.3) How effective (or non-effective) has ACE been in improving WASH? (IR 1.4) How effective (or non-effective) has ACE been in increasing community capacity for preparedness, mitigation, and responses to disaster? (IR 1.5) Prompt: Can you give an example of how this approach impacted your grant or activity?

EQ3: To what extent has a multi-sectoral and integrated community development approach contributed to achieving expected outcomes, intermediate results, and objectives in the areas of:

a) Reducing community vulnerabilities b) Strengthening community participation, and; c) Strengthening local governance?

8 As you may know, ACE was designed to be multi-sectoral and integrated in its approach, bring CSOs and CBOs together with local government to improve community welfare. To what extent do you think this happened?

Note: Based on the type of involvement the interviewee has explained above, chose the most appropriate prompt question.

9 How has ACE’s multi-sectoral and integrated community development approach helped or hindered its ability to strengthen community participation?

Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in improving community capacity and leadership skills? (IR2.1) Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in improving participatory community decision-making? (IR 2.2) Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in developing community action plans? (IR 2.3)


# Questions Guidance, Adaptions, & Prompts Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in enhancing trust and reconciliation? (IR 2.4) Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in increasing youth and women’s participation? (IR 2.5)

10 How has ACE’s multi-sectoral and integrated community development approach helped or hindered its ability to strengthen local governance?

Note: Based on the type of involvement the interviewee has explained above, chose the most appropriate prompt question. Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in integrating public service provision? (IR 3.1) Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in strengthening the local public service provider’s capacity? (IR 3.2) Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in increasing trust and collaboration between communities and local authorities? (IR 3.3)



EQ 4: How has the organizational structure of the ACE activity (a combination of direct implementation, consortium partners, and sub-partner grants) affected ACE’s ability to achieve desired outcomes and results at this point in the project implementation?

a) From the perspective of the prime implementing partner b) From the perspective of the consortium partners c) From the perspective of the local partner d) From the perspective of the direct beneficiaries

2) The challenges to coordination when in a consortium different member report up to different managers and response to different sets of performance targets.

3) The positive contribution of having a shared office to aiding coordination and program implementation;

4) How coordination among partners can be held back when they are moving at different speeds, and,

5) 4). The challenges to coordination when partners are working in completing different fields (health, education, agriculture) and some partners came to the project with long established relationships in the southeast (CPI and SC)

11 From your perspective, how has the way ACE has been organized been effective in delivering your activity and results or hindered the implementation of your activities and achieving results?

[Note: Responding to this question depends on the respondent noting a familiarity with ACE in the introductory questions. If limited familiarity with ACE’s structure, skip this question.] Prompt: What could have or should have been done differently? Prompt: What should be done differently in the next two years or beyond? (EQ 6)

EQ 5: How has the COVID-19 pandemic: a) affected ACE’s ability to meet expected outcomes, intermediate results, and

objectives? b) resulted in the modifications to respond to the needs which arose out of the

pandemic? 12 How has COVID-19 affected your organization’s ability to

implement its activities? How do you understand ACE had been impacted?

Prompt: How has this changed in the last six months? Prompt: How do you expect it to change in the future? (EQ6) Prompt: What has been the impact of the pandemic on your international or local partners?



What modifications have you had to make to continue your activities? What were those changes and why were they made?

Prompt: How has this changed in the last six months? Prompt: How do you expect it to change in the future, if at all? (EQ6)

Have the objectives, targets, interventions, and/or populations of interest for your organization changed because of COVID-19? Please explain.

Prompt: Has anything else changed?

How has your organization adapted or been unable to adapt to COVID-19?

Prompt: Have there been bureaucratic, funding, or political pressures? Please explain. About frustrations of staff at the implementation level about their frustration with USAID's restrictions or inertia on doing COVID-related activities?

EQ 6: Where are the potential opportunities to alter, scale-up, reinforce, or cancel activities or approaches over the remainder of the program in response to the changing context and political dynamics in the implementation areas? 13 [Prompt: Read EQ6 to the interviewee]

Do you have any specific recommendations for the projects last two years or beyond?

Prompt: Specifically, what do you see as potential opportunities to:

a) alter b) scale-up, c) reinforce, or; d) cancel activities or

approaches


KEY INFORMANT INTERVIEW ACE PROJECT STAFF



Interview Date:



Respondent Sex:


What is your ethnicity? [Ask respondent to self-identify the ethnicity]



Respondent Region:

Start Time:  

End Time:  

Hello, I am _______________ and I work with Social Impact, an independent company conducting research for the United States Agency for International Development (USAID) Burma Office. We are conducting an evaluation of the Advancing Community Empowerment in Southeastern Myanmar (ACE) project. We intend to interview approximately 40 key stakeholders in this project, and you have been selected to participate because we believe you can provide a unique perspective on the work of ACE.


Your personally identifiable information, such as your name and phone number, will not be connected to any information you provide in this interview. The data from this interview will be used by the Social Impact team for analysis purposes only and your answers will only be reported as part of an aggregated data set. This report will be shared with ACE and made publicly available on USAID DEC. To ensure all your responses are captured accurately and can be reviewed during data analysis and report writing, we would like to record the interview. The recordings uploaded to a password protected site and will not be made public.

This interview will last approximately one hour. Do you understand that your participation in this interview is voluntary, unpaid, and that you can end the interview at any point? If you have any follow up questions, please contact Myint Zaw at 09450064401 or Christine Thomas at [email protected]. You may also reach the Social Impact IRB in the U.S. at [email protected] or +1 703-465-1884.






# Questions Guidance, Adaptions & Prompts

1 Can you describe your role within your organization and your involvement with the ACE project?

Prompt: How long have you been in this position and what ways have you been directly involved with ACE? Prompt: What responsibilities do you have with regards to ACE? Prompt: What activities are you directly involved with regards to ACE? Prompt: Did you have any role in the design of this project?





4 How have these changes impacted ACE and your activities within the project?

Note: Separate prompts depending on role/activity of partners or grantee Prompt: How have the changes impacted health care access and coordinated service delivery? (IR 1.1) Prompt: How have the changes impacted improving education access and services? (IR 1.2) Prompt: How have the changes impacted improving sustainable livelihoods? (IR 1.3) Prompt: How have the changes impacted improving WASH? (IR 1.4)


# Questions Guidance, Adaptions & Prompts Prompt: How have the changes impacted increasing community capacity for preparedness, mitigation, and responses to disaster? (IR 1.5)




Note: Separate prompts depending on role/activity of partners or grantee Prompt: How have the changes impacted increasing the integration of public service provision? (IR 3.1) (EQ3) Prompt: How have the changes impacted strengthening local public service provision? (IR 3.2) (EQ3)


# Questions Guidance, Adaptions & Prompts Prompt: How have the changes impacted increasing trust and collaboration between communities and local authorities? (IR 3.3) (EQ3) Prompt : What is the impact of the turnover of govt officials in key positions at the township level on program implementation? Prompt : Which change has a greater impact on the ground? What are some examples of where changes in local govt personnel have had negative or positive impacts on project implementation?


a) Health care access and coordinated service delivery improved b) Education access and services improved c) Sustainable livelihoods promoted d) WASH improved e) Community capacity for preparedness, mitigation and responses to disaster

increased? 7 What contribution does your activity or organization

make to ACE’s goal of reducing community vulnerabilities?


How effective (or non-effective) has ACE been in improving health care access and coordinated service delivery? (IR 1.1) How effective (or non-effective) has ACE been in improving education access and services? (IR 1.2) How effective (or non-effective) has ACE been in promoting sustainable livelihoods? (IR 1.3)


# Questions Guidance, Adaptions & Prompts How effective (or non-effective) has ACE been in improving WASH? (IR 1.4) How effective (or non-effective) has ACE been in increasing community capacity for preparedness, mitigation, and responses to disaster? (IR 1.5) Prompt: Can you give an example of how this approach impacted your grant or activity?








# Questions Guidance, Adaptions & Prompts Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in enhancing trust and reconciliation? (IR 2.4) Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in increasing youth and women’s participation? (IR 2.5)




a) From the perspective of the prime implementing partner b) From the perspective of the consortium partners c) From the perspective of the local partners d) From the perspective of the direct beneficiaries




Prompt: What could have or should have been done differently? Prompt: What should be done differently in the next two years or beyond? (Also, EQ 6)



pandemic? 12 How has COVID-19 affected ACE’s ability to implement

its activities, meet expected outcomes, intermediate results, and objectives?

Prompt: How has this changed in the last six months? Prompt: How do you expect it to change in the future? (EQ6) Prompt: What has been the impact of the pandemic on the activities of the VCG / VDC / VHC / VWC / CAN or VD who your work with? [Note: Choose relevant body, if applicable]

Has COVID-19 resulted in modifications to ACE project activities? What were those changes and why were they made?

Prompt: How has this changed in the last six months? Prompt: How do you expect it to change in the future, if at all? (Also, EQ6)

Have the objectives, targets, interventions, and/or populations of interest have changed because of COVID-19? Please explain how?


How has your organization or ACE activity adapted or been unable to adapt to COVID-19?


EQ 6: Where are the potential opportunities to alter, scale-up, reinforce, or cancel activities or approaches over the remainder of the program in response to the changing context and political dynamics in the implementation areas?



13 [Prompt: Read EQ6 to the interviewee] Do you have any specific recommendations for the projects last two years or beyond?



approaches



KEY INFORMANT INTERVIEW LOCAL PARTNERS



Interview Date:



Respondent Sex:


What is your ethnicity? [Ask respondent to self-identify the ethnicity]


Respondent Region:

Start Time:  

End Time:  

Hello, I am _______________ and I work with Social Impact, an independent company conducting research for the United States Agency for International Development (USAID) Burma Office. We are conducting an evaluation of the Advancing Community Empowerment in Southeastern Myanmar (ACE) project. We intend to interview approximately 40 key stakeholders in this project, and you have been selected to participate because we believe you can provide a unique perspective on the work of ACE.


Your personally identifiable information, such as your name and phone number, will not be connected to any information you provide in this interview. The data from this interview will be used by the Social Impact team for analysis purposes only and your answers will only be reported as part of an aggregated data set. This report will be shared with ACE and made publicly available on USAID DEC. To ensure all your responses are captured accurately and can be reviewed during data analysis and report writing, we would like to record the interview. The recordings uploaded to a password protected site and will not be made public.

This interview will last approximately one hour. Do you understand that your participation in this interview is voluntary, unpaid, and that you can end the interview at any point? If you have any follow up questions, please contact Myint Zaw at 09450064401 or Christine Thomas at


[email protected]. You may also reach the Social Impact IRB in the U.S. at [email protected] or +1 703-465-1884.




1 Can you describe your role within your organization and your involvement with the ACE project?

Prompt: How long have you been in this position and what ways have you been directly involved with ACE? Prompt: What responsibilities do you have with regards to ACE? Prompt: What activities are you directly involved with regards to ACE? Prompt: Did you have any role in the design of this project?





4 How have these changes impacted ACE and your activities within the project?

Note: Separate prompts depending on role/activity of partners or grantee Prompt: How have the changes impacted health care access and coordinated service delivery? (IR 1.1)




# Questions Guidance, Adaptions & Prompts Prompt: How have the changes impacted improving education access and services? (IR 1.2) Prompt: How have the changes impacted improving sustainable livelihoods? (IR 1.3) Prompt: How have the changes impacted improving WASH? (IR 1.4) Prompt: How have the changes impacted increasing community capacity for preparedness, mitigation, and responses to disaster? (IR 1.5)




Note: Separate prompts depending on role/activity of partners or grantee Prompt: How have the changes impacted increasing the integration of public service provision? (IR 3.1) (EQ3)


# Questions Guidance, Adaptions & Prompts Prompt: How have the changes impacted strengthening local public service provision? (IR 3.2) (EQ3) Prompt: How have the changes impacted increasing trust and collaboration between communities and local authorities? (IR 3.3) (EQ3) Prompt : What is the impact of the turnover of govt officials in key positions at the township level on program implementation? Prompt : Which change has a greater impact on the ground? What are some examples of where changes in local govt personnel have had negative or positive impacts on project implementation?


a) Health care access and coordinated service delivery improved b) Education access and services improved c) Sustainable livelihoods promoted d) WASH improved e) Community capacity for preparedness, mitigation and responses to disaster

increased 7 What contribution does your activity or organization

make to ACE’s goal of reducing community vulnerabilities?


How effective (or non-effective) has ACE been in improving health care access and coordinated service delivery? (IR 1.1) How effective (or non-effective) has ACE been in improving education access and services? (IR 1.2)


# Questions Guidance, Adaptions & Prompts How effective (or non-effective) has ACE been in promoting sustainable livelihoods? (IR 1.3) How effective (or non-effective) has ACE been in improving WASH? (IR 1.4) How effective (or non-effective) has ACE been in increasing community capacity for preparedness, mitigation, and responses to disaster? (IR 1.5) Prompt: Can you give an example of how this approach impacted your grant or activity?








# Questions Guidance, Adaptions & Prompts Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in enhancing trust and reconciliation? (IR 2.4) Prompt: How effective has ACE’s multi-sectoral and integrated community development approach been in increasing youth and women’s participation? (IR 2.5)




a) From the perspective of the prime implementing partner b) From the perspective of the consortium partners c) From the perspective of the local partners d) From the perspective of the direct beneficiaries




Prompt: What could have or should have been done differently? Prompt: What should be done differently in the next two years or beyond? (Also, EQ 6)



pandemic? 12 How has COVID-19 affected ACE’s ability to implement

its activities, meet expected outcomes, intermediate results, and objectives?

Prompt: How has this changed in the last six months? Prompt: How do you expect it to change in the future? (EQ6) Prompt: What has been the impact of the pandemic on the activities of the VCG / VDC / VHC / VWC / CAN or VD that you work with? [Note: Choose relevant body, if applicable]

Has COVID-19 resulted in modifications to ACE project activities? What were those changes and why were they made?

Prompt: How has this changed in the last six months? Prompt: How do you expect it to change in the future, if at all? (Also, EQ6)

Have the objectives, targets, interventions, and/or populations of interest have changed because of COVID-19? Please explain how?


How has your organization or ACE activity adapted or been unable to adapt to COVID-19?


EQ 6: Where are the potential opportunities to alter, scale-up, reinforce, or cancel activities or approaches over the remainder of the program in response to the changing context and political dynamics in the implementation areas?



13 [Prompt: Read EQ6 to the interviewee] Do you have any specific recommendations for the projects last two years or beyond?



approaches

MINI-SURVEY INSTRUMENT/PROTOCOLS

Recruitment Script [Script to be translated into Burmese and all local languages used for the survey.]

Background: This script is for use by ACE staff and village coordinators to, if necessary, introduce Social Impact enumerators to ACE direct beneficiaries.

Note: Script can be used orally or as a text message:

• Social Impact is an independent company conducting research for the USAID’s Burma Office that funds ACE. The purpose of this evaluation is to assess the progress of the project and identify best practices and lessons learned.

• They are conducting a survey of around 150-200 direct beneficiaries and you have been randomly selected to take part in this survey. This survey will take approximately 20 minutes. and will cover activities undertaken in your community by ACE.

• Participation is voluntary. You can decline to take part in the survey without any changes to the support provided by ACE to your community, however, the answers that you give can help improve ACE in the next two years.

• Your answers will be anonymous/anonymized and ACE will not have access to the answers you provide. Your name or any identifying information will not appear in the survey’s results or the final evaluation report.

Consent question: Do you agree to be contacted by the Social Impact survey team?

[If yes] Thank you, they will be in touch with you shortly to set up the interview.

[If no] This is not a problem. Thank you.

Survey instrument Target Groups:

● Village Coordination Groups (VCG)

● Village Development Committee (VDC)

● Village Health Committees (VHC)


● Village Water, Sanitation, and Hygiene (WASH) committees (VWCs)

● Villages that have conducted Community Needs Assessments (CNA)

Informed Consent and Confidentiality:

Hello, I am _______________ and I work with Social Impact, an American independent company conducting research for the United States Agency for International Development (USAID) Burma Office. We are conducting an evaluation of the Advancing Community Empowerment in South-eastern Myanmar (ACE) project. We intend to interview approximately 150-200 beneficiaries in this project, and you have been randomly selected to participate.

Your participation in this interview should not pose any harm or discomfort to you. Indeed, your participation in this interview is completely voluntary and you are free to decline to respond to any question for any reason and to end the interview at any time, with no consequence to you or your community. You will not be provided with any payment for your participation in this interview. Your honest responses will help USAID understand the results of ACE.

Your name will not be connected to any information you provide in this interview. The data from this interview, including information like your age, village, and participation in groups, will only be seen by the Social Impact team for analysis purposes. The information will be kept by us on our team’s computers and will be kept confidential to the extent allowed by local and US law and USAID policy. The Social Impact team will use information provided by you and other beneficiaries of ACE to create a report about our findings. This report will be shared with ACE and made publicly available on USAID DEC. This interview will last approximately 20 minutes. Do you understand that your participation in this interview is voluntary, unpaid, and that you can end the interview at any point?

If you have any follow up questions, please contact Myint Zaw at 09450064401 or reach the Social Impact IRB in the U.S. at +1 703-465-1884.

Do you confirm your consent to participate in this survey?

1. Yes

2. No [If no, terminate the interview.]

Is this a good time for you to speak with me and are you able to speak comfortably and privately? If not, I can arrange to call you at another time.

1. Yes

2. No. [ If no, schedule a call-back time]

Do you consent to this interview being recorded to ensure your answers are correctly noted?

1. Yes

2. No. [If no, note answers manually without audio recording.]


No. Question Responses Code/Instruction

ID1 I Interviewer 1. Add interviewer code into the Mobile app

ID2 Survey Date (don’t ask respondent) --/--/2020

ID3 State/Region: (don’t ask respondent) 1. Kayin

2. Kayah

3. Mon

4. Shan

ID4 Township: (don’t ask respondent) 1. Demoso

2. Hlaingbwe

3. Hpasawng

4. Hpruso

5. Kyainseikgyi

6. Pekon

7. Thanbyuzayat

8. Thandaunggyi

ID5 Village Tract/ ward; (confirm with respondent)

To put VT name in the mobile app

ID6 Villages/ ward (confirm with respondent) To put village name in the mobile app

Q1 Respondent code

Q2 Survey Respondent Sex 1. Female

2. Male

Q3 Age of Respondent ________



Q4 Name of committee / group (confirm with respondent)

VGC/VDC

1. VHC/WASH

2. Education/PTA

3. WORTH

Q5 Position in the committee / group Add positions in the Mobile app

Q6 What is your role and responsibilities in this organization or group?

First, we want to ask you some questions about the structure of your organization or group (VGC/VDC/VHC/VWC) or the group that conducted the VNA:

Q7 How many members are there in your organization or group?

-----

Q8 How many women are members of your organization or group?

-----

Q9 How many young people (under 29 years) are members of your organization or group? (estimated number)

-----

Q10 How were you selected to be in your position in this group?

1. Nominated by self

2. Nominated by peers

3. Chosen by village/group leaders

4. Chosen by project staff

Other ____

Q11 Did this group exist before the ACE project started?

1. Before

(Pre-existing Name: _____________)

2. Created by ACE

If (2), skip “after column”



If they respond “before,” they please ask her/him to compare before and after ACE became involved with the group.

a. Before

b. After

Q12 Before the COVID-19 pandemic, how often does your group hold a meeting?

1. At least weekly

2. At least monthly

3. At least once every three months

4. At least yearly

5. Never

6. Other ____

Q13 Since the COVID-19 pandemic restrictions started, how often has your group held a meeting?

7. At least weekly

8. At least monthly

9. At least once every three months

10. At least yearly

11. Never

12. Other ____

Q14 What are key activities of your organization or group?

1. ________

2. ________

3. ________

Q15 What does your organization or group typically discuss in your meetings?

1. ________

2. ________

3. ________

Q16 How often do you personally take part in the discussion during these meeting?

1. Very often

2. Often

3. Sometimes

4. Seldom

5. Never



Q17 How are decisions made in your organization or group?

1. By leader

2. By vote

3. By consensus

4. Other

5. Decisions not made

Q18 How often are decisions made by your organization or group implemented after the meeting?

1. Very often

2. Often

3. Sometimes

4. Seldom

5. Never

Can you tell me about the most important action your group or organization has taken in the past year?

Q19 What action? Who raised the issue?

Who proposed the action/solution?

Who made the decision on the action?

A.

B.

C.

D. No response No response No response No response

Post-survey questions for enumerator

PS1 Which language was used for this interview?

PS2 Did you feel the respondent had been coached or prepared for this survey?

1. Yes

2. No.

PS3 If yes to PS 1, please briefly describe why you believe this to be the case.


ANNEX IV: DISCLOSURE OF ANY CONFLICTS OF INTEREST

[The Evaluation Policy requires that evaluation reports include a signed statement by each evaluation team member regarding any conflicts of interest. A suggested format is provided below.]

Name

Title

Organization

Evaluation Position? Team Leader Team member

Evaluation Award Number (contract or other instrument)

USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable)

I have real or potential conflicts of interest to disclose.

Yes No

If yes answered above, I disclose the following facts:

Real or potential conflicts of interest may include, but are not limited to:

1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated.

2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation.

3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project.

4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated.

5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated.

6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation.

I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other


companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished.

Signature

Date


U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT

1300 PENNSYLVANIA AVENUE, NW

WASHINGTON, DC 20523

MID-TERM EVALUATION REPORT

Documents

Transcript of MID-TERM EVALUATION REPORT