Chapter 1 Overview of ODA EvaluationChapter 1 Overview of ODA Evaluation 4 called for the promotion...

Chapter 1

Overview of ODA Evaluation

1.1 Recent Trends in ODA Evaluation ■

3

Chapter 1 Overview of ODA Evaluation

1.1 Recent Trends in ODA Evaluation

The importance of evaluation of official development assistance (ODA) has gradually increased since its commencement in 19751), as a means of improving the effectiveness and efficiency of ODA and as a means to meet the requirement for accountability to the public. In response to changes in the environment for ODA in Japan as well as overseas, and also as the culmination of a series of ODA reforms, the ODA Charter was revised in 2003. The “Formulation and Implementation of ODA Policy” section of the revised Charter cites an enhanced evaluation as one of the necessary factors for effective ODA implementation.

The discussions on ODA evaluation in the international community began in the 1970s at the Development Assistance Committee (DAC) of the Organization of Economic Cooperation and Development (OECD) and successive conferences on development financing. A full-scale effort began in the 1980s with the establishment of the evaluation correspondents group under the DAC. With a series of changes and subsequent reorganization of the DAC in 2003, the DAC Working Party on Aid Evaluation was reorganized into the DAC Network on Development Evaluation in order to work toward the realization of more effective and efficient development assistance with close collaboration of the newly established Working Party on Aid Effectiveness and Donor Practices. 1.1.1 Revision of the ODA Charter and the Enhancement of ODA Evaluation (1) Revision of the ODA Charter

The ODA Charter approved by a cabinet meeting in 1992 laid down the basic philosophy, principles and priorities of ODA and had been the fundamental document of Japan’s ODA policy for more than a decade. During this decade, however, the international scene underwent dramatic changes and ODA needed to respond to a new set of challenges, including the issues of worsening poverty brought on by globalization, escalation of regional or internal conflicts and international terrorism in the post-Cold War world. The environment surrounding ODA is changing day by day, with the emergence of new assistance modalities, such as those provided by NGOs, universities and research organizations, businesses and local public organizations, as well as new techniques such as the Comprehensive Development Framework (CDF) and the Sector Wide Approaches (SWAPs).

Faced with severe fiscal constraints and increasingly critical public view of ODA, the Ministry of Foreign Affairs (MoFA) has been actively involved in the government effort toward ODA reform measures anchored in the key concepts of transparency, efficiency and public participation. The MoFA engaged in various actions for reform, beginning with the establishment in 2001 of the Second Consultative Committee on ODA Reform, a Foreign Minister’s private consultation group, and in 2002 of the Advisory Board on Organizational Reform of the Ministry of Foreign Affairs, which discussed the reform of the Ministry itself. The ODA reform efforts culminated in August, 2003 with cabinet approval of the new ODA Charter. The revised ODA Charter states that “the objectives of ODA are to contribute to the peace and development of the international community and, thereby to help ensure Japan’s own security and prosperity”. The new Charter emphasizes mobility, strategy, transparency, efficiency and public participation in ODA, as well as the promotion of better understanding of ODA.

(2) Enhanced evaluation in the new ODA Charter

The former ODA Charter stipulated that evaluation activities be expanded to include third-party evaluation and joint evaluations with other countries for the benefit of future foreign aid. The Charter also

1) Initiated by the then Overseas Economic Cooperation Fund (OECF, now the Japan Bank of International Cooperation) in 1975. The MoFA began its ex-post evaluation in 1981, and JICA followed it in 1982.

■ Chapter 1 Overview of ODA Evaluation

4

called for the promotion of comprehensive evaluation of ODA. These stipulations reflected the situation that existed when the former ODA Charter was formulated in 1992. At that time, the MoFA, overseas diplomatic offices, implementation agencies and third-parties were conducting ex-post evaluation. The third-party evaluation, however, accounted for only a small part, and broadening this area was being called for. In addition, although the MoFA was trying to measure the effectiveness of Japan’s foreign aid using the results of evaluation of individual projects, the basic policies governing those projects were not evaluated. Under this circumstance, many were pointing out the importance of more comprehensive evaluation.

The former ODA Charter (excerpt relating to evaluation)

4. Measures for the Effective Implementation of Official Development Assistance (10) Cooperation and research to find and formulate adequate development projects

will be enhanced. For the future improvement of ODA, project evaluations, including third party evaluations and joint evaluations with recipients and other donors and organizations, will also be strengthened.

(11) Regional studies of developing countries, studies of development policies, and comprehensive evaluation of ODA will be further promoted.

Since then, the significance and scope of ODA evaluation had changed substantially by the time the ODA

Charter was revised in August 2003. In the past, ODA evaluation was conducted in parallel with the planning and implementation procedures. As, however, the idea of a cycle of planning (Plan), implementation (Do) and evaluation (See) for effective management of government activities became established, the ODA evaluation began to be incorporated as an integral part of the ODA system. The fact that attention was given in the ODA sector to the adoption of new government management practices, such as the New Public Management (NPM) and the Result-Based Management (RBM) in North America and Europe may have been an influential factor in this development. In Japan, the Government Policy Evaluation Act (the “GPEA”), which came into effect in 2002, requires all administrative organizations to evaluate their own policies for the purposes of promoting effective public administration and ensuring accountability to the public.

Furthermore, the introduction of comprehensive development approaches that transcended individual projects, such as the CDF and the SWAPs, called for the expansion of the scope of evaluation from individual projects to assistance activities in a specific sector or even in a country-level.

Based on such background, the revised ODA Charter identified evaluation as a necessary factor for the effective implementation of assistance, and expressed its attitude to expand its evaluation activities.

One of the most prominent features of the new ODA Charter is the effort to implement a comprehensive evaluation from the perspectives of both timing and scope. Specifically, the evaluation system covers each stage of ODA activities, from ex-ante to ex-post, and covers a wider spectrum with policies at one end and individual projects at the other. The incorporation of ex-ante and mid-term evaluations into the planning and implementation processes of ODA made an integrated evaluation possible, whereas, in the past, the evaluation was conducted on an individual project basis and only after its completion. In an effort to enhance its effectiveness, the focus of development assistance has shifted in recent years to a more comprehensive approach, which targets a sector or a country as a whole in addition to individual projects. An evaluation with broader scope becomes possible by targeting not only individual projects but also assistance activities at sector or country level.

In addition to the expansion of third-party evaluation, another feature of the new Charter concerning the significance of evaluation is the mention on the self-evaluation by government agencies. The expansion of third-party evaluation were stipulated in the former ODA Charter, and were cited as a pillar of evaluation system reform, essential for enhancing the objectivity of evaluation in the series of ODA reform. Furthermore, the aforementioned enforcement of the GPEA in 2002 required the MoFA to evaluate its ODA projects as part of its policy evaluation. For these reasons, the new ODA Charter refers to both types of evaluation, broadening also the scope of evaluators.


5

The new ODA Charter also stipulates that evaluation results should be reflected in the formulation and implementation of policies. The evaluation is meaningful only when its results are used effectively as feedback in the planning and implementation of policies. While the former ODA Charter did not expressly state this point, the new Charter points toward the importance of having evaluation results reflected in the subsequent ODA activities.

The ODA Charter of 2003 (excerpt relating to evaluation)

III. Formulation and Implementation of ODA Policy 3. Matters Essential to Effective Implementation

(1) Enhancement of evaluation The government will carry out consecutive evaluation at all stages, i.e. ex-ante,

mid-term, and ex-post, and evaluation at each level, i.e. policy, program and project. Furthermore, in order to measure, analyze and objectively evaluate the outcome of ODA, third-party evaluations conducted by experts will be enhanced while the government undertakes the policy evaluations. The evaluation results will be reflected in subsequent ODA policy-making and efficient and effective implementation.

(3) Measures for the enhancement of evaluation and future challenges

The new ODA Charter established the course of action for the expansion of the evaluation system. As will be described later, the MoFA has been making steady progress in implementing the requirements of the new Charter to expand the third-party evaluation, by changing the organization of the ODA evaluation and conducting studies on policy- and program-level evaluations. The evaluation, however, is still in its changing phase, and there are issues that still need to be addressed. The MoFA continues to make efforts toward improvements in this regard.

First, the expansion of the scope of evaluation to include policies and programs has presented technical problems. Fairly well established techniques are available for the evaluation of individual projects because of its long history. The Project Cycle Management (PCM) method currently employed by JICA is one of them. On the other hand, there is no established technique not even internationally, for policy- and program-level evaluations, and individual donor countries and international organizations are working on it by trial and error. The MoFA has adopted the method of comprehensively evaluating ODA from the points of view of objective, process and effectiveness, but the improvement of methods for assessing the results of ODA remains a challenge. With respect to the verification of the results of ODA, a technique called Performance Measurement has been adopted, especially in Western countries. This method provides a set of policy indicators at the formation stage of the plan, to determine the policy goals and to measure how much they have been accomplished, and monitors regularly the changes in these indicators. This technique is effective when the causality between individual ODA activities and their results are easily definable. If, however, the evaluation aims at a country or a sector as a whole, there is difficulty in setting plausible goals. At present, the MoFA is studying the feasibility of introducing indicators for ODA activities.

In recent years, the importance of joint monitoring and evaluation with recipient countries is gaining recognition, in response to active adoption of comprehensive development approaches, such as the PRSP (Poverty Reduction Strategy Paper), which emphasizes the ownership of development by recipient countries. This is based on the viewpoint that joint evaluation, rather than individual evaluations by donors, can reduce the burden on the recipient countries, and that participation of recipient countries in the evaluation will contribute to enhanced ownership. Although this approach has been introduced in the evaluation of programs for Tanzania and a few other countries to date, it is still at its early stage, and its effectiveness needs to be ascertained and problems identified. In addition, the MoFA has been discussing issues relating to evaluation with Asian countries, other donors and international organizations at various international conferences. For example, the MoFA hosted the Third Tokyo Workshop on ODA Evaluation in November 2003, the main topic of which was joint evaluation. (See Column 1.)


6

The evaluation is meaningful when its results are then used in the formulation and implementation of policies. To this end, the quality of evaluation results needs further improvement. At present, the Internal Study Meeting on ODA Evaluation established under the authority of the Director of Economic Cooperation Bureau of the MoFA discusses and determines the measures needed to respond to the evaluation results. The MoFA is making efforts to conduct more in-depth studies with the participation of its other bureaus and divisions concerned so that the evaluations may provide more realistic and specific recommendations.

The evaluation results give feedback to the relevant parties, and are published so as to ensure accountability to the public and promote a better understanding of ODA. In this connection it is worth mentioning that a seminar was held in 2003 to present the evaluation results of the Okinawa Infectious Diseases Initiative (IDI). (See Column 2.)

Column 1 The Third Tokyo Workshop on ODA Evaluation

The MoFA has been conducting a workshop on ODA evaluation since FY2001 for the purpose of strengthening the relationship between donors and recipient countries in Asia. The Third Tokyo Workshop on ODA evaluation was held at the Keio Plaza Hotel over two days, November 12 and 13, in FY2003. 1. Background

The First Workshop was held in 2001 with the objective of promoting a common understanding of ODA evaluation. Representatives from Asian countries and international organizations attended the meeting, and discussed the assistance in legal and institutional fields necessary for effective ODA evaluation. The participants agreed on the recognition of the importance of involvement in ODA evaluation of partner countries (i.e. recipient countries) and their capacity building. The Second Workshop held in 2002 focused on the role of evaluation in new assistance modalities. The discussions pointed out the importance of policy- and program-level evaluation, in addition to individual project evaluation. 2. Objectives

The Third Workshop pursued the following objectives, based on the discussions of the past two workshops, as well as on the discussions at the DAC Working Party on Aid Evaluation and at the ministerial meetings of the Initiative for Development in East Asia (IDEA)2): • To reinforce the shared understanding that ODA evaluation is an important tool for effective development

assistance; and • To identify the issues relating to joint evaluation by a donor and a recipient country, and discuss future

approaches to deal with these issues. 3. Agenda

Based on the objectives above, the agenda of the workshop focused mainly on discussions on the role of evaluation and on joint evaluations. The outcomes of the discussions were summarized by the chairperson of the workshop, Dr. Ryokichi Hirono, Prof. Emer., Seijo University, as the Chairperson Summary (see next page) • Agenda 1: Monitoring and evaluation in the management of international development goals: Status of

monitoring and evaluation of MDGs. • Agenda 2A: Joint evaluation 1:

Issues for joint monitoring/evaluation in Asia - Country/Sector-level Approach Issues for joint monitoring/evaluation in Asia - Project-level Approach

• Agenda 2B:Joint evaluation 2: Future tasks for joint monitoring and evaluation

2) The basic goals of the Initiative for Development in East Asia are to promote economic collaboration and regional cooperation in East Asia. Its objectives include a shared recognition of the current situation regarding developmental issues in East Asia, and exploring the future direction of development cooperation in the region. The first IDEA Conference was held in Tokyo in August 2002, attended by ministers of foreign affairs as well as ministers in charge of development and education of the ASEAN countries.


7

4. Participants The participants included representatives from eighteen Asian countries (Bangladesh, Brunei, Cambodia,

China, India, Indonesia, Korea, Laos, Malaysia, Mongolia, Myanmar, Nepal, Pakistan, Philippines, Singapore, Sri Lanka, Thailand and Vietnam), a DAC member (France), as well as representatives of the Asia Development Bank (ADB), the World Bank, the DAC/OECD, the United Nation Development Programme (UNDP), and the United Nations Children’s Fund (UNICEF)

Workshop on ODA Evaluation

by Prof. Hirono, Chairperson of “The 3rd Tokyo Workshop on ODA Evaluation”

12-13 November 2003

Japan hosted “The Tokyo Workshop on ODA Evaluation” on 12-13 November 2003 at Keio plaza hotel. This was the third workshop on ODA Evaluation following the last two years.

Fifty participants including senior government officials from eighteen Asian countries, evaluation specialists of a donor country and multilateral development agencies, Japanese government officials of ODA-related ministries, staff of Japan International Cooperation Agency (JICA), Japan Bank for International Cooperation (JBIC) and academic persons also attended this workshop. A number of people attended the workshop in an observer status.

The participants recognized the importance of donor-partner cooperation to tackle development challenges and global development issues. To enhance effectiveness of development and cooperation, it is necessary to set out a long-term comprehensive development strategy and a management system. Through this workshop, the participants expressed their shared recognition of the importance of donor-partner cooperation based on partner countries’ ownership. In particular, joint donor-partner monitoring and evaluation was recognized as an effective tool for management of development assistance and overall development activities.

Participants from Asian countries, together with bilateral and multi-lateral organizations, have discussed and shared their knowledge and experiences on the methods to manage ODA activities. While recognizing the critical importance of a comprehensive development approach, the participants, based on their experiences, exchanged their views mainly on how to manage ODA more effectively, and on challenges and difficulties in conducting joint evaluation.

In each country, the general public is increasingly concerned with transparency, accountability and participation in conducting government activities. In this respect, it was significant that this workshop was open to the public.

Chairperson's Summary


8

All participants appreciated Japan’s initiative to organize the third workshop for the purpose of deepening understanding of the importance of monitoring, evaluation and collaboration among Asian countries and the donor community.

Some highlights of the discussions include: It is necessary to streamline a management cycle of planning, implementation, monitoring and evaluation

for maximum development impact in partner countries. The key is how to integrate monitoring and evaluation systems into the whole development process for greater effectiveness of development and cooperation. Recognizing the importance of country ownership and multiplicity of donors, it is important that the scope of evaluation should cover the sector- and country-level in addition to the project level. The transaction costs3

are generally reduced when donor and partner cooperate within a common framework, in spite of some difficulties.

What is necessary to tackle the development challenges through collaboration between donors and partner countries is to agree on common development goals. Stakeholders including government, local bodies, private sector and civil society should be involved, wherever appropriate, in the planning process. Also, we need to consider how donor countries and organizations align with the partner country's priority development goals. Coordination should be made in a flexible and country-specific manner. In order to use the limited resources more efficiently, long-term comprehensive strategic goals are essential. In this connection, achieving MDGs poses a great challenge.

In managing development activities, monitoring and evaluation indicators should be set to assess to what extent the objectives are achieved. More reliable quantitative indicators are needed while the importance of qualitative indicators should not be underestimated. Although outcome-based evaluation is important, the assessment of process and secondary impact should not be overlooked. Moreover, the ability to collect and analyze data should be enhanced for regular measurement of the set indicators.

The effectiveness of development activities is improved when evaluation results are utilized effectively. It is, therefore, essential to establish feedback mechanisms so that the evaluation results are incorporated into the planning and implementation process. For this purpose, development organizations are encouraged to make active use of evaluation results.

The key is capacity building in partner countries. There have been many practices of donor-partner and South-South cooperation for human resource development. The workshop provided an opportunity for exchange of knowledge and experiences among donors and partners in monitoring and evaluation. The workshop recommended that, in addition to the establishment of an evaluation association in each country, evaluation networks should be strengthened among Asian countries. The workshop welcomed the proposal made by Thailand to host a joint evaluation seminar next year and the proposal by Vietnam to conduct a joint monitoring and evaluation exercise, with possible impact on capacity building. In this connection, the participants welcomed the initiative of the Japan Evaluation Society in following up these efforts in cooperation with other similar organizations.

The report of this workshop will be published soon. The participants from Asian partner countries are kindly expected to utilize the results of the workshop for further improvement of their monitoring and evaluation systems.

3) Transaction cost: The cost of transaction activities of goods and services, which must be borne by the participants in the transaction.


9

Reconstruction Plan of the Engineering Department of East Timor University (East Timor)

A scene from the Outreach activities (Cambodia)


10

Column 2 ODA Evaluation Seminars

The MoFA works in the enhancement of evaluation as part of its ODA reform and holds ODA Evaluation Seminars as part of its effort in publishing the evaluation results, with the purpose of ensuring its accountability to the public and promoting the public understanding on ODA.

On March 24, 2004, the MoFA, jointly with JICA and JBIC, held the ODA Evaluation Seminar, “Japan’s Contribution to World Pandemic of Infectious and Parasitic Diseases: Mid-term Evaluation Study of Okinawa Infectious Diseases Initiative (IDI) and Japan’s Future Efforts”. 120 people attended the meeting held at the international conference hall of the Institute for International Cooperation of JICA, including representatives from Diplomatic Corps and international organizations.

In the first part of the seminar, keynote speeches were presented on the assistance for measures against AIDS, tuberculosis and malaria, and on the efforts taken by the Japanese government. In the second part, the Tuberculosis Prevention Society presented the results and recommendations of its mid-term evaluation study of the IDI, commissioned by the MoFA in FY2003. The study assessed the IDI launched by the Japanese government at the G8 Kyushu/Okinawa Summit in July, 2000 from the perspectives of whether its purpose were relevant (relevance), whether its processes of formulation and implementation were appropriate (appropriateness) and whether the IDI was effective (effectiveness). The report acknowledged the enormous significance of the IDI in raising the world political interest and setting off the global fight against infectious diseases, but pointed out the lack of specific strategies for assistance and of domestic personnel to develop the strategies.

In response to the above items pointed out by the mid-term evaluation results of the IDI, the panel discussion in the third part of the seminar, “International Contribution of Japan on Measures against infectious diseases,” focused mainly on human resources development and strategic approaches for assistance to measures against infectious diseases. The panel recognized the importance of human resources development both in the prevention of infectious diseases in Japan as well as in international cooperation, and it was pointed out that it is necessary to have personnel who can manage the program as a whole (i.e. international health experts) as well as experts in individual diseases. Furthermore, the panel proposed that human resource development be strengthened by organic linkage between research budgets of the Ministry of Health, Labour and Welfare and the Ministry of Education, Culture, Sports, Science and Technology and the “Junior Professional Officers (JPO)” of the MoFA. With respect to strategic approaches, it was pointed out that there should be more opportunities for meaningful discussions as this seminar, and that an “all-Japan” system needs to be built, encompassing the MoFA, the Ministry of Education, Culture, Sports, Science and Technology, the Ministry of Health, Labor and Welfare, JICA, JBIC and NGOs, in order for Japan to establish a cohesive strategic approach. The panel discussion was moderated by Professor Yasuhide Nakamura of Osaka University, advisor for the Mid-term evaluation study of IDI. The panelists are listed below, and the discussion was open for questions and comments from the audience. • Naofumi Yasuda, Medical Engineering Officer, Bureau of

International Cooperation, International Medical Center of Japan

• Nobukatsu Ishikawa, Vice President, The Research Institute of Tuberculosis, Japan Anti-Tuberculosis Association

• Tsutomu Takeuchi, Professor, School of Medicine, Keio University

• Osamu Kunii, Deputy Director, Research & Programming Division, Economic Cooperation Bureau, MoFA

• Akira Hashizume, Deputy Manager and Acting General Manager, Medical Cooperation Dept., JICA

• Takeo Matsuzawa, Deputy Director General, Project Development Department, JBIC

• Nobuhiro Kadoi, Senior Program Officer, Japan Organization for International Cooperation in Family Planning

ODA Evaluation Seminar for IDI


11

1.1.2 The DAC Network on Development Evaluation (1) Reorganization and mandates

Since the establishment of the Evaluation Correspondents Group in the 1980s to discuss ODA evaluation, the DAC has been establishing organs specialized in evaluation over time. In conjunction with the reorganization of the DAC in 2003, which aimed at enhancing efficiency, the Working Party on Aid Evaluation was modified as the DAC Network on Development Evaluation. (“The Network”)

The Network is comprised of representatives of about 30 bilateral donor countries and international organizations, and is chaired by the Netherlands4). The mandate of the Network is to: share knowledge of evaluation, and strengthen cooperation on evaluation; contribute to the improved effectiveness of development assistance; provide advice to DAC; and assist recipient countries in developing their evaluation capacity. The Network meets twice a year.

The mandate of the DAC Network on Development Evaluation

1. Strengthen the exchange of information, experience and cooperation on evaluation among Network members and, as appropriate, with development evaluation partners, with a view to: • improving the evaluation activities of individual members; • encouraging harmonization and standardization of methodological and conceptual

frameworks; • facilitating co-ordination of major evaluation studies; • encouraging development of new methods in evaluation and best practice.

2. Contribute to improved development effectiveness by: • synthesizing and extracting policy, strategic and operational lessons from

evaluations for consideration by the DAC and the wider development community; • promoting joint or co-ordinated evaluations and studies undertaken by individual

members. 3. Provide advice and support to DAC and its subsidiary bodies, notably on peer reviews,

development results and aid effectiveness. 4. Promote and support evaluation capacity development in partner countries.

(2) The First Meeting of the DAC Network on Development Evaluation

The Network held its first meeting in January 2004. Issues facing evaluation were discussed and each member reported their major evaluation activities.

The discussion on the issues facing evaluation centered on quality. The members agreed that minimum standards needed to be established so as to ensure a minimum level of quality. As for the setting of specific standards, however, they agreed to continue with the discussion, as it relates to the issues of reliability and transparency of evaluation. The agenda for the focus session was dedicated to joint evaluation methods, taking into account the increasing adoption of comprehensive development strategies, such as the Poverty Reduction Strategy Papers. Participants agreed on the importance of conducting joint evaluation in step with the development cycle of the recipient countries and coordinated by donor countries. On the other hand, the discussion pointed to obstacles, such as time lag, attribution of assistance results among donors, and the lack of data necessary for evaluation.

With respect to the major evaluation activities, the members presented reports on evaluation of general budget support and their collaborative efforts with recipient countries. General budget support provides funds directly into the general budget of the recipient countries, and is one of the important forms of assistance under the Poverty Reduction Strategy Papers. Currently, sixteen countries and organizations, including Japan, participate in the evaluation of general budget support under the leadership of Great Britain

4) Effective January 2004. Sweden will assume the chair as from the second meeting scheduled for November 2004.


12

and the Netherlands. The status of development of the guidelines currently under way toward the full-scale implementation of evaluation studies on this form of assistance was reported at the January, 2004 meeting of the Network. With respect to collaboration efforts with recipient countries, highlights from the evaluation seminar hosted by France as well as the Third Tokyo Workshop on ODA Evaluation were presented and well received by the members in attendance.

(3) Future agenda

As noted above, the Network is working together to share knowledge on evaluation and improve the effectiveness of development assistance. The next major agenda include studies on techniques for country- or sector-level comprehensive evaluation and for joint evaluation, as well as review of evaluation quality. The importance of evaluation, both independently or jointly, of development assistance for a country or a sector as a whole will increase along with the deployment of comprehensive development approaches emphasizing the ownership of the development project by recipient countries. With a view to such a movement, Japan will continue to actively participate in these activities.

1.1.3 Monitoring of the Millennium Development Goals (MDGs)

The MDGs are a single common framework that put together the development goals set by the United Nations Millennium Declaration as well as other major international conferences and summit meetings held in 1990s. The MDGs provide the international community with guidelines for the formulation of development policies and programs. They also provide the common framework to measure the effectiveness of these policies and programs. The MDGs consist of eight goals and eighteen targets. The United Nations, OECD, IMF and the World Bank are supposed to regularly assess the progress toward the achievement of these goals. The United Nations monitors the indicators set in the MDGs at the global-level as well as at the country-level. The results of the global-level monitoring are included in the UN Secretary-General’s Report, while those on the country level are included into the Millennium Development Goals Reports (MDGRs). The MDGRs are reports on the trends in the indicators of recipient countries, and are prepared in fifty countries at present. In addition, multilateral development banks, which includes the World Bank, is monitoring the progress toward the MDGs, and reports its results to development financing meetings. Also, a mid-term review of the Millennium Declaration, including the MDGs, is scheduled for 2005, and with it, monitoring activities are expected to increase in the near future.

1.2 Organization of ODA Evaluation ■

13

1.2 Organization of ODA Evaluation

Japan’s ODA evaluation has been conducted primarily by the MoFA and by the implementing agencies (i.e. JICA and JBIC) since it began in 19755). For an efficient evaluation, the MoFA and the implementing agencies share the evaluation work according to their respective functions. The Basic Law for Central Government Reform (1998) gave the MoFA a pivotal role in the coordination of overall ODA planning in the entire government. In response to this new function, the MoFA has been organizing the Experts Meeting of ODA Evaluation, consisting of the ministries and agencies involved in ODA. The MoFA has been organizing evaluation results by these ministries and agencies. 1.2.1 Role of the MoFA and the implementing agencies

In order to ensure efficient evaluation of ODA, the MoFA and the implementing agencies give importance to mutual cooperation and the division of roles. Since the MoFA is responsible for the planning and development of economic cooperation policies, it focuses its evaluation on policies and programs rather than individual projects. Conversely, JICA and JBIC are responsible for implementing or facilitating the implementation of individual projects, and accordingly, they focus their evaluation on these projects. The policy-level evaluation of ODA assesses the basic ODA policy. Accordingly, the MoFA has been evaluating medium-term policies on ODA, as well as country policies and priority issues. The program-level evaluation targets groups of projects that share common objectives, while the project-level evaluation reviews individual activities such as single projects.

Fig. 1 Organization and Objects of ODA Evaluation

5) The ministries and agencies involved in ODA also evaluate their own ODA projects.


14

Column 3 The Experts Meeting of ODA Evaluation and ODA evaluation by the ministries and agencies

The Inter-Ministerial Liaison Meeting on ODA Evaluation was established in July, 2001 for the main purpose

of promoting collaboration of the ministries and agencies involved in ODA. The Meeting was subsequently renamed to the “Experts Meeting of ODA Evaluation” in November 2002. The Meeting has held discussions on standards and guidelines for evaluation of ODA projects and has been promoting the collaboration of the ministries and agencies.

The Meetings continued in FY2002 to discuss what the ODA guidelines should be and how ODA evaluation result should be compiled. As a result, Annual Evaluation Report on Japan’s Economic Cooperation, published in March 2003, included for the first time the results of the evaluations carried out by various ministries and agencies, outlining them in Chapter 4.

The ODA projects carried out by various ministries are roughly divided into the categories of: acceptance of trainees, dispatch of experts, research and donations to international organizations. Evaluations are done on the project, the program- and policy-level. Evaluations are conducted at ex-ante, mid-term or ex-post stage, depending on the project, and may sometimes cover a mixture of ODA and non-ODA projects.

This report also compiles a wide range of evaluation reports, selected based on flexible criteria, outlining the ODA evaluations by the ministries and agencies in a simple way. (See Section 2 of Chapter 2 for the evaluation results. The details of each evaluation are available at the website of individual ministries or agencies.)

The MoFA will continue with discussions at the Experts Meeting of ODA Evaluation in its effort to present the overall state of the ODA evaluation in a more comprehensible manner. 1.2.2 ODA Evaluation by the MoFA (1) Purposes of ODA evaluation

The ODA evaluation is required to function as a means for the effective and efficient implementation of ODA as part of the government’s activities. It also provides the public as tax payers with information on ODA. To these ends, the MoFA conducts its ODA evaluation with the following two purposes: • Support for ODA management

To support the management of ODA and the improvement of its quality through the examination of ODA activities and feedback of the lessons learned into the planning and implementation processes of ODA policies.

• Accountability To ensure accountability to the public and enhance transparency of ODA through publishing the results

of the evaluation so as to promote public understanding of and public participation in ODA. (2) Functions of evaluation

In order to meet these requirements, the MoFA conducts an evaluation at each of the ex-ante, mid-term and ex-post stages of the ODA implementation cycle6). The ODA evaluation results at each stage give feedback to ODA policymakers and implementation agencies (both in Japan and in the recipient countries) and are utilized as a useful tool in ODA management. Also, to ensure accountability, the ODA evaluation results are made available to the public on the Internet through the MoFA website.

6) Monitoring of the basic policies are currently under consideration.


15

Fig. 2 Functions of ODA Evaluation

(3) Type of evaluation

The ODA evaluation by the MoFA can be classified in various types. In accordance with the object of evaluation, it can be classified into policy-level, program-level and project-level evaluation7). In accordance with evaluators, it can be classified into third-party evaluation, evaluation by the MoFA, evaluation by government/Organization in Recipient Country, and joint evaluation. Of these, evaluation by the MoFA is mandated by the GPEA that requires the government organizations and agencies to evaluate their own activities8).

A policy-level evaluation assesses a set of programs and projects aimed at realizing Japan’s basic economic cooperation policies. There are two kinds of policy-level evaluation: country policy evaluation and priority issue evaluation. Country policy evaluation reviews assistance policies toward individual recipient countries. Specifically, it focuses on ODA country policies and Country Assistance Plan. The priority issue evaluation looks at initiatives for priority issues announced by the Japanese government at summits and other international conferences. One such example of evaluation object is “Japan’s Initiative on Women in Development (WID)” announced at the Fourth World Conference on Women in 1995.

A program-level evaluation examines a group of projects that share common objectives. It is divided into the sector program evaluation and type of aid evaluation. A sector program evaluation basically looks at a

7) The MoFA made a substantial review of the types of evaluation in FY2002, following the expansion of ODA evaluation. 8) The MoFA conducts ex-post evaluation on its policies for specific countries/regions and sectors as required by the GPEA. The evaluation of ODA-related policies is part of these evaluation activities. In addition, the GPEA requires the MoFA to conduct evaluation on unstarted or unfinished projects as well as ex-ante evaluation of the project evaluations. The results of these evaluations are complied together with the results from other departments within the MoFA and organized into an evaluation report, which is published at the same time it is submitted to the Ministry of Internal Affairs and Communications


16

group of ODA activities in a particular sector of a particular country, while a type of aid evaluation examines a single type of ODA.

A project-level evaluation focuses on an individual project. It is carried out mainly by the implementing agencies (i.e. JICA and JBIC). The MoFA also conducts project evaluations from a policy perspective, pursuant to the requirements of the GPEA. Specifically, the MoFA is required to conduct ex-ante evaluation of all grant aid projects valued over one billion yen and loan aid projects (i.e. yen loan projects) over fifteen billion yen9). The MoFA also conducts project-level evaluations on projects which are not yet implemented after a certain time has passed following the policy decision (i.e. unstarted projects) and those not yet completed (unfinished project).

Table 1 Types of ODA Evaluation by the Ministry of Foreign Affairs

Level Name Evaluator Object Timing Policy-level Country policy

evaluation • Third-party evaluation; • Joint evaluation; • Evaluation by the MoFA*

ODA Country Policy • Mid-term• Ex-post

Priority issue evaluation

• Third-party evaluation; • Joint evaluation; • Evaluation by the MoFA*

ODA Policy on Priority Issues & Sectors

• Mid-term• Ex-post

Program-level Sector program evaluation

• Third-party evaluation; • Evaluation by

governments/organizations in recipient country ;

• Joint evaluation

Sector Development Program or a set of projects in specific sector

• Mid-term• Ex-post

Type of Aid evaluation

• Third-party evaluation; • Evaluation by

governments/organizations in recipient country ;

• Joint evaluation

Specific form of Japan’s assistance

-

Project-level Project evaluation

• Evaluation by the MoFA* Individual projects • Ex-ante • Ex-post

* Evaluation required by the GPEA (4) Organization and flow of evaluation

Of the above types of ODA evaluation used by the MoFA, the Economic Cooperation Bureau of the MoFA takes charge of the third-party evaluations, evaluations by the governments or organizations in recipient countries and joint evaluations, and the Minister’s Secretariat coordinates the evaluations carried out by the MoFA under the GPEA.

The MoFA emphasizes third-party evaluations for ensuring objectivity of the ODA evaluations. In the past, these third-party evaluation studies were in principle carried out by consultants, but in order to further enhance objectivity, it is delegated to the External Advisory Meeting on ODA Evaluation as of October, 2003. This Meeting is private council under the authority of the Director of the Economic Cooperation Bureau, and consists mainly of outside experts. Besides, in some cases, the government or agencies (e.g., think tanks and academic organizations) of recipient countries are asked to conduct evaluation so that their perspectives are reflected in the results. In other cases, the MoFA conducts evaluations jointly with external agencies (e.g., other donors, recipient countries or NGOs).

9) The implementing agencies (JICA and JBIC) conduct ex-ante evaluations to assess the appropriateness and the estimated results of individual projects. The results are published in the Ex-Ante Project Evaluation Sheet. In contrast, the ex-ante evaluations by the MoFA assess projects from a policy perspective.


17

Fig. 3 Organizations of ODA Evaluation by MoFA

Basically, these evaluations follow the process of planning, implementation, feedback and publication.

First, the MoFA prepares a 3-year Medium-term evaluation plans on ODA evaluation as well as an annual plan and determines the type and the object of evaluation. Next, evaluators are selected according to individual objects of evaluation who determine the basic plan for their evaluation study. In the actual evaluation study, specific evaluation methods are decided according to this basic plan, followed by survey in Japan, a field study and formulation of report based on analysis of the findings. The evaluation result gives feedback to ODA policymakers and implementation agencies /personnel, and is made public through website, among other means.

Fig. 4 Flow of the Third-Party Evaluation

As the government organizations and agencies are required to evaluate their own policies under the GPEA

enacted in 2002, the MoFA implements evaluation on ODA activities as well. Divisions in charge within the Economic Cooperation Bureau, conducts the evaluation according to the ministry’s policy evaluation plan. The results are recorded in the prescribed evaluation sheet, and submitted to the Policy Evaluation and Administrative Review Office of the Minister’s Secretariat. In addition to the reports from the Economic Cooperation Bureau, this Office collects evaluation sheets from various other bureaus and division of the ministry. After reviews by various organizations involved in evaluation (i.e. Policy Evaluation and Administrative Review Office, the Management and Coordination Division/Financial Affairs Division of the Minister’s Secretariat, and the Policy Coordination Division/the Policy Planning Division of the Foreign Policy Bureau), the results are organized as a policy evaluation report, which is then submitted to the Ministry of Internal Affairs and Communications as well as published on the Internet through the MoFA’s website10). The Ministry of Internal Affairs and Communications compiles the evaluation reports submitted by various government ministries and agencies into a report and submits it to the Diet.

10) The MoFA’s policy evaluation is available at http://www.mofa.go.jp/mofaj/annai/shocho/hyouka/index.html.


18

Column 4 The 2003 Report on Evaluation Feedback Activities of the External Advisory Meeting on ODA Evaluation

by Hiromitsu Muta, the External Advisory Meeting on ODA Evaluation

Responding to the new ODA Charter that provides for the expansion of third-party evaluation by experts, the

Wise Men Committee for Evaluation Feedback was commissioned by the Director of the Economic Cooperation Bureau of the Ministry of Foreign Affairs to conduct ODA evaluations, and was renamed in October, 2003 the External Advisory Meeting on ODA Evaluation. Following these organizational changes in the MoFA’s ODA evaluation, the Meeting conducted the following evaluation studies in FY2003: Country program evaluation on Jordan; Evaluation on Japan Disaster Relief; and Cultural Grant Aids. In other evaluation studies, the members of the Meeting were involved in the development of the evaluation methods at an early stage, and in some of them, the Committee also participated in field studies.

The Meeting also conducted its conventional feedback activities, and provided comments on the MoFA’s measures in response to the recommendations contained in evaluation reports. The MoFA takes these comments into account when it determines its measures at the meetings of the Internal Study Meeting on ODA Evaluation Review set up within the Economic Cooperation Bureau (renamed from the Internal Feedback Liaison Meeting on ODA Evaluation in accordance with organizational changes mentioned above; includes representatives of JICA and JBIC). The implementation status of the measures are then reported to the External Advisory Meeting. Below is the outline of the evaluation feedback activities of the External Advisory Meeting in FY2003. 1. Evaluation studies examined by the External Advisory Meeting on ODA Evaluation

Third-party evaluations commissioned by the MoFA in FY2002, including the following: (1) Country Policy Evaluation Study on Sri Lanka (2) Country Policy Evaluation Study on Thailand (3) Evaluation on the WID/Gender-related Policy (WID Initiative) (4) Evaluation on the South-South Cooperation (5) Evaluation of the NGO Subsidy Program (a joint evaluation with NGOs) 2. Major Comments by the External Advisory Meeting on ODA Evaluation (1) The reports on the country policy evaluation on assistance to Sri Lanka and Thailand recommended that

indicators should be established at the planning stage of the assistance policies and schemes, in order to measure the degree of goal accomplishment. The comments of the External Advisory Meeting on ODA Evaluation were that: indicators should be specified during the formulation of the assistance plan. It is thought that it is possible to monitor the changing trends in relation to the assistance goals. Although not all indicators need to be quantitative, it should be possible to adopt practical indicators, including semi-qualitative indicators which can be answered simply with yes or no.

(2) The evaluation report on the assistance policy for Sri Lanka recommended that in order to enrich the contents of the ODA country policy, it should include commentaries based on the experience of implementing agencies and experts as well as suggestions for assistance methods that enhances the self-sustainable development of recipient countries. The commentaries may suggest, for example, what the public administration of the recipient country should be (such as the need for collaboration among government organizations and agencies involved). The comment of the Meeting was that: The recommendations in the report regarding how the public administration should be and the need for self-sustainable development are issues relating to governance. Accordingly, the response measures by the MoFA should refer to the need for assistance for the issue of governance.


19

(3) Although not directly related to the MoFA’s measures in response to the recommendations in the evaluation report, the Meeting commented, with respect to the evaluation of the WID/Gender-related Policy, that: The WID Initiative emphasized health, education and participation of women in society. These tended to narrow down the sectors for assistance. It is important to shift the emphasis toward gender equality from a cross-sectional perspective of ODA.

(4) The report on the evaluation of the South-South cooperation recommended that, based on the fact that the South-South cooperation contributes to the reduction of ODA cost, it is beneficial to increase the number of participating countries in the partnership program, taking into account the implementation capacity, including cost-bearing capacity, of cooperating countries. The comments of the Meeting was that: It may not be necessary to expect a proportionate cost-sharing if the purpose of the program is cost reduction. Upon receiving an explanation from the MoFA that part of the intention of the partnership program in the South-South cooperation was to change the mindset of recipient countries from that of a beneficiary to a benefactor, the Meeting’s comment was that: Given that, such political intention should be reflected in the response measure to the recommendation. In addition, the Meeting pointed out that, the MoFA should pay attention to the need for capacity building in cooperating countries as noted in the evaluation report. In response to this comment, the MoFA described a case in Mexico that made a concomitant effort toward the capacity development of the cooperating country.

(5) Additional comments were that: The recommendations in a country policy evaluation report should specify whether they are applicable only to the country under evaluation or whether they are general recommendations; As a general argument, it is essential that factual implementation be accurately collected in an evaluation. Accordingly, it is important to develop an evaluation framework that ensures consistency of evaluation results regardless of who the evaluators are, and establish an evaluation manual based on it.

Reference - The Members of the External Advisory Meeting for ODA Evaluation:

Hiromitsu Muta Professor, Tokyo Institute of Technology (Chairman) Teruo Kawakami Certified Public Accountant (KPMG AZSA&Co.) Hiroshi Sato Senior Fellow, IDE-JETRO Yasunaga Takachiho Professor, Tamagawa University Tomoya Masaki Visiting Professor, Graduate School, The University of Tokyo Yasuko Muramatsu Professor, Tokyo Woman's Christian University

Chapter 1 Overview of ODA EvaluationChapter 1 Overview of ODA Evaluation 4 called for the promotion...

Documents

Transcript of Chapter 1 Overview of ODA EvaluationChapter 1 Overview of ODA Evaluation 4 called for the promotion...