214 UI World Bank Discussion Papers

From MacroeconomicCorrection to PublicSector Reform

The Critical Role of Evaluation

Eduardo Wiesner D.

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Recent World Bank Discussion Papers

No. 156 IDjelopiviA.q ricaluaral lixtevsionfor 14alltevaIirivvven. Ktnnie A Saito and lDplane Spurling

No. 157 Awvakeniqedivite Market: l'ierr arn ' IonoEn rma. Tra:jimo. 1). M. Lcipzijgr

No. 158 IiVfae Policyd untai' die 7rav-sitiotir to a a.vrket lEovalovv': l'oLm /d 1990- 91. Fabrizio Coricelt aiad All.i lCvrnga.

ed itors

No. 15') hitevvativaoal fadreavnd filit liv'irovvienevl. I'arnck Lotw. 'itor

No. 1() intrernvatiotial.Migrationand Ivar,ith 'evnil ri-laral'd Sharon StantOtn IU%CII andIMiciao S. -rCnCithJIIIi

No. 161 ComiISenicu RekJrn and tire 11 ;rld ibtvk Blarbtr. NulI ergnandJhn Nel

No. 162 RIIral luerpnpe Deavelov"nctI an (Ititah. In'J19r- A9 Anriluyiv3. ()dv

No. 163 71Te Balance bcfu'eeva Public and Priaure Sectr'Amirvifres Ima t- Delivery iflai'c,fock Serrto.n. Dins L. U,aajl,. ( ;crt.lii

Fedcr. and Cornclis de Haan

No. 164 lIcni Do i\ational :\l.;a.,-, .lfet' 1vrig-nrn (Gyotd'dt? .A Re.eranh .Agecnda Wi\Vllaiii Latcrlv. Robert King. ILU%%Levine. and Scrgio Rebelo

No. 165 Fishene5 DneeLopprent. IF imen, Mlarivaagent7ar. aind Lnrenjahncs Rtichiard S. Johlnstoni

No. 6 77t1 e Biling Bb u' ck, ofPaiarricipnan: livingl. lit'tom-u;- Pla Michael NI. Ccrnca

No. 1(7 Seed Systemt Dt'lcrepinetir: 77i A4pprolna:r Rolles. of lith l4ai'.z:c arin Puli, Scto rt. Stvceniatlt' Cand latctadra Sriv.itrava

No. I S 6 Enair. ovnenvaal . Iarlan.eniernr aivil U 'rbva idneraliltry. Alcira Krenicr and iMhnIls] Muniisinghe. edirom

No. 169 Cornnion Property Resources: A .1 1)s in Dimenswon oJ zlvl-iek lrnn it SrrnaergcS N. S -Jozdia

No. 171) A Chinese Pira'inc as a Rclk;nn lipcn rncn: 7fTe Ci;c otf I arinan PaIUI N. Cadano. azukko OJaswa. a.ld Yzt-K.nnl Wcn

No. 171 las nesorir Inriastrncture.Alanaagcvenew n tle 1990,. Arturo lsracl

No. 172 Japanese Ntovianal Rail'ays Prizatiov Staid): fie Exptiriuce oI Japanadi I ..es,oslor DIh,'lopivag Couivtrne -

Koichiro Fukui

No. 173 nhe Li'esrock Seror in Eastern Em rope: Covsrrairts and OJworturairie. Comelis de Haart. Tjaart Schilihonm V.an

Vccn. and Karen Brooks

No. 174 .Assessivng Dea'elopwne fintance listirattioni: A IPublic Iritere,t Analysis. Jacob Yaron

No. 175 Resource Maiaaevvneritantd Pasorarl hIstitiaon BuildirIn In rhe lf`tsr .fricav .Sahelt Nadarajah Shanmugaratnain. Trond

Vedcld. Anie Mossige. and Mctte Bovin

No. 176 Ptblic and Piatel: Sector Roles uti Ayraaultaaral Researlh: Theory and Experience. D)ula L. Until

No. 177 'Die Revdlatory Ivapeditnevris to0 the %irate Induistmal Seaor Deiopiznentr in Asia: A Ctomniparatir .Stddy. D)oela Khackharc

No. 17S Clina: Refonning lhiergovemnivental Fiscal Relarions. Rkarngopal Agarvala

No. 179 Nippon Telegraph anid Telephone Prinatizarioni Study: lExperirce oljapan and lx.,sona .1r Dlerilopiavrg Gisii.rn

Yoshiro Takano

No. 181 Chinia s Refonn Experience to Date. Peter Harrold

No. 1I 1 Cnbatrirng AIIDS avid Orher Sexually Transvtritted Diseawses Africa: A Re'iew .y'rfie 111irld

Jean-Louis Laniborav and A. Edward Elmendorf

No. 182 Pri,'atization Problevns ar industry Itevl: Road Iaulage in Ceitral Europe. Esra Bennathan and Louis S. Thompson

No. 183 Participatory Development arid the UWo rld Bank: Potential Direcons fiJr Change. Bhuvan Bhatnlagr and Aubrey C.

Williams. editors

Io:..vrintied on the inside back cover.)

2 14 E 1 World Bank Discussion Papers

From MacroeconomicCorrection to PublicSector Reform

The Critical Role of Evaluation

Eduardo Wiesner D.

T-he World BankWashigon, D.C.

Copyright O 1993The International Bank for Reconstructionand Devc1opment/THE WORD BANK

1818 H Street, N.W.Washington, D.C. 20433, U.S.A.

All rights reservedManufactured in the United States of AmericaFirst prnting August 1993

Discussion Papers present results of country analysis or research that is circulated to encourage discussionand comment widtin the development community. To present these results with the least possible delay, thetypescript of this paper has not been prepared in accordance with the procedures appropriate to formalprinted texts, and the World Bank accepts no responsbility for errors

The findings, interpretations, and condusions expressed in this paper are entirly those of the author(s) andshould not be attributed in any manrer to the Wodd Bank, to its afiliiated orgaizations, or to members ofits Board of Executive Directors or the countries they represent. The World Bank does not guarantee theaccuracy of the data induded in this publication and accepts no responsibility whatsoever for anyconsequence oftheir use. Any maps that accompany the text have been prepared solely for the convenienceof readers; the designations and presentation of material in them do not rmnply the expression of any opinionw hatsoever on the part of the World Bank, its affiliates, or its Board or member countries concering thelegal status of any country, territory, city, or area or of the authorities thereof or concerning the delimitationof its boundaries or its national affiliation.

The matcrial in this publication is copyrihtecL Requests for permission to reproduce portions of it shouldbe sent to the Office of the Publisher at the address shown in the copyright notice above. The Wodd Bnlkcncourages dissemination of its wodc and will normally give permission prompdy and, wnen thereproduction is for noncommercial purposes, without asking a fee. Permission to copy portions for classroomuse is grnted through the Copynrght Clearnce Center, 27 Congssr Saln Massachusetts 01970, USA

The complete backist of publications ftom the World Bank is shown in the annual Idex of Pulions,

which contains an alphabetical tide list (with fall ordering information) and indexes of subjects, authors, andcountrics and regions. The latest cdition is available free of charge from dte Distrbution Unit, Office of thePublisher, The World Bank, 1818 H Street, N.W., Washington, D.C. 20433, U.SA, or from Publications,The World Bank, 66, avenue d'I6na, 75116 Paris, France.

ISSN: 0259-210X

Eduardo Wiesner D. is a consultant to the World Bank's Office of the Director General, OperationsEvaluation.

Library of Conges Cataloging-in-Publicadon Data

W'iesner Durin, Eduardo.From macroeconomic correction to public sector reform: the

critical role of evaluation / Eduardo WiesnerD.p. cm. - (World Bank discussion papers ; 214)

Includes bibhographical references.ISBN 0-8213-2650-31. Developing countries-Economic policy. L Title. H. Series.

HC59.7.W514 1993338.9'009172'4-dc2O 93-31849

CIP

ABSTRACr

This paper posits that, a decade after the debt crisis bepn, pubic sector reform hasbecome the central developmental challge: once a developing country achieves roughmacroeconomic balance, recvety and sustainable growth depend on achieving 'institutionaladjustmenL' Technical asstance has not been evey effectivc in inducing public sector reform,partly because it has often aimed at tansferring sklils instead of building domestic capacity.This paper explores the proposition that private sector deveopment demands a restucturedpublic sector and that evaluation capacity buildig holds a privieged place within the variouscomponents of public sector reform. It then sketches a strategy for increased multilateralassistance to enhance development learing through evaluation.

...

CONTS

PAge No.

FOREWORD vii

ACKNOWLEDGMENTS viii

EXECUE SUMMARY ix

INTRODUCTION 1

L EVALUATION CAPACIIY EN DEVELOPING COUNT 3

a. Ile Meaing and Objectives of Evaluation 3

b Requie and Cnstraints 4

C. The Objectives of Evaluation Capacity in Developing Countries S

IL EVALUATION CAPACIYf, PUBUC SECTOR REFORM AND GOVERNANCE INDEVELOPING COUNI 8

a. Public Sector Reform 8

b. Governance and Evalution Capacity 11

mL TWE POITCAL ECONOMY OF EVALUATION CAPACITV IN DEVELOPINGCOUNTRIES 15

a. Guiding Princpls fir Refrm 15

b. Normative Recommdio and Positive Realities 18

v

IV. THE INSTITUTIONAL FRAMEWORK FOR EVALUATION CAPACITYDEVELOPMENT IN DEVELOPING COUNTRI 20

a. The Enabling Environment for Evaluation Capacity Development 20

b. The Institutionalization of Evaluation Capacity 21

c. Altemative Approaches and Restictions 22

d. The Role of the Private Sector 23

V. THE STRATEGY FOR EVALUATION CAPACITY DEVELOPENT INDEVELOPING COUNTIE AND THE ROLE OF MULTIATERALINSTITIUIONS 24

a. Multilateral Fnance and Aid: A Public Good? 24

b. Intnational Political Eonomy and Governance 26

C. A Multiateral Straty in Support of Devcklpng Countrues' Efforts 27

VL CONCLUDING REARKS 30

NOTES 35

BBIJOGRAP CAL REFENCES 39

vi

FOREWORD

The World Bank has long provided advice on strengthening evaluation capacities to itsmember countries. There has been a recent increase in the demand for such advice, and thediscussion paper that foiws provides a conceptual framework that goes a long way inexplaining this development.

Demand for evaluation is linked to popular demand for transparency and accountability.As decentralzation spreads and greater reliance is placed on the private sector, public sectorskills are being increasingl shifted towards public expenditures management and policyevaluation by goverments intent on ensuring that the lessons of ecperience are used toimprove the efficiency of public expenditures. Thi disuassion paper is likely to promotedemand for evaluation in situations where policy makers are inhibited from investingadministrative resources in evaluation by outdated notions of public sector reform priorities.

Eduardo Wiesner D, a former Executive Director of the World Bank and a formerMinister of Fmance of Colombia, has first-hand experience in development financing andimplementation. He prepared this dicuion paper in connection with the InternationalWorkshop on Evaluation Capacity Building held at World Bank headquarters in July, 1993.The paper trigered sprited debates among the participants. Its publication should contributeto an even wider debate about the roe of evaluation in enhancing development effectiveness.

Robert PicciottoDirector GeneralOperations Evaluation

vi

ACKNOWLEDGEMENTS

Ihis paper has benefitted from the comments and criticisms of several people who readit thoroughly and found reasons to agree and to disagree I would like to thank, in particular,John Erikuson, Robert Picciotto, Gregory Ingram, Pablo Guerero, Mark Baird, Harinder Kohli,Peter Richardson, Annette Bimnendijk, Nina Vredland, Mervyn L Weiner, Jean S. Quesnel, andC Ray Rist Thanks are also due to Angela Gentie-Blackweil who provided editorialassistance. Needless to say, I remain solely responsible for the findings, interpretations andconclusions eessed in this paper.

Eduardo Wiesner D.

Vi;i

EXECUTIVE SUMARY

For many developing countries, one of the most striking legacies of the debt crisis hasbeen the discovery that public sector reform and institutional development appear to be thecritical link between macroeconomic correction and sustainable growth. In effect, theirexperience indicates that while a modicum of macroeconomic balance is a necessary conditionfor growth, it is not sufficient on its own, and that the restructuring of the public sector maywell be the key factor for the attainment of durable growth This poses a new and formidablechallenge. After all, such public sector reform involves voluntary "public choices' to change therole of governments and to modify the existing balance of power and interests.

This paper argues that the strengthening of evaluation capacity (EC) is the best answerto such a challenge. Developing countries (DCs) wishing to move from macroeconomiccorrection to faster and more sustainable growth will find in evaluation capacity building thepath to a more efficient public sector, and thus to a more productive economy as a whole.Compared to other means, evaluation seems to be the most powerfid instrument of publicsector reform in the sense that it is more effective in revealing the causes of "governmentfailure' and in creating an environment for its correction. Evaluation brings mechanismi adprocedures into governmental interventions to test their effectiveness. In this context, publicsector reform is tantamount to the correction of "government failure."

Tfhe importance of public sector reform has not gone undetected. Assistance forinstitutional development from the World Bank, for example, has risen as a proportion oftechnical assistance at a time when technical assistance has also grown relative to total lending.This support, however, does not seem to have had the desired impact (World Bank 1991).While the reasons are partially explained by the intrinsic complexity of institutionaldevelopment, it may well be that insufficient focus on one single strategic front, such asevaluation, is also part of the answer. This does not mean that other efforts to reform publicsectors, ie., divestiture, decentralization of decision making, and the fostering of efficiencythrough competition, should be abandoned. It means, however, that since resources are scarce,it would be more effective to concentrate on the enhancement of evaluation capacity than tomaintain a dispersed effort across the whole spectrum of public sectors. In briet er postevaluation offers a strategy for public sector reform.

The development of EC will not be easy or feasible as a quick fix for public sectors inDCs. EC involves a technical dimension as well as a political one. The analytical logic for itsdeployment will be insufficient for its political viability, and it would be naive to always assumethat the participating DC wil have a benign and welfare-maximizing government whose purposeis to apply policy nostrums in pursuit of the public good. And yet, many developing countrieshave adopted a more market-oriented strategy, and are seeking to reform their public sectors

ix

I

and to make their governments more politicayl accountable. There is thus a propitious climateand, at the same time, there are political obstacles. The strengthening of EC is at the center ofeconomic and political reform. What precisely triggers the right "public choices" for EC tobecome official policy is not easily determined. In most cases, policy results from a combinationof factors including political leadership, demonstration effects, information and the dynamics ofchange brought about by abysmally incompetent public sectors.

One way to circumvnt the inherent limitations of "public choices' is to pose the policyissues in terms of global, structural and constitutional reforms that appear remote and, in thewords of J. Buchanan (1991), 'behind a veilr which disguises some of their real implications.Giving independence to central banks, eliminating ear-marked funds, or balancing the budgetmay be included in these kinds of reforms. In Colombia, for instance, a 1991 constitutionalreform mandated that a system of evaluation be developed in the public sector. As a result,Colombia stands well-poised to develop a national evaluation system.

Once a country has decided to launch an EC program, one of the questions that arisesrelates to the strategic placement of the main evaluation office or unit. The answer seems to bethat the Ministry of Fmance, or wherever budget decisions are nade, on aUl levels ofgovernment, is the preferable place to institutionalize evaluation. his facflitates compliancewith the sine qua non condition of evaluation, Le., that there be adequate feedback ofevaluation results into current and future policies. Indeed, the key to successful EC building isto constantly search for an active link and interplay between evaluation results and budgetdecision making.

The enabling environment for EC development should include:

(1) a macroeconomic model and a development strategy that ismarket-oriented,

(2) an open political system in which information can flow freelyand bring about greater political accountability and improvedgovernance;

(3) public demand that policies and priorities be determined by anopen political exchange; and

(4) political demand that public funds be used more efficiently.

A strategy for the development of evaluation capacity in DCs and for the role ofmultilateral institutions should begin with three global considerations. Fist, there is a 'two-wayrelationship" in that external lending and aid affect evaluation capacity in DCs, and thedevelopment of EC in these countries in tuan influences: (1) the level and characteristics oflending by multilateral institutions, and (2) the 'in-house' evaluations of operations. A secondconsideration to keep in mind is that multilateral lending is often perceived as a special type of"public good" by some DCs and even by some of the providers of such resources. This affects

Id

the EC relationship between DCs and multilateral institutions. A third global factor that needsto be considered is that just as there is a 'national political economy," there is also an*international political economy,' the institutional environment in which lending takes place, aidis donated, and debtor and creditor countries try to evaluate what they all are doing. Thisinternational political economy environment conditions the relationships between the differentparties and, in particular, their efforts to be more efficient and accountable.

The specific insight that this paper offers is that evaluation capacity development byDCs, multilateral institutions and aid agencies could be the framework to bring about higherefficiency and accountability to alL EC could also contnbute to the formulation of moreconsistent intemational policies and to greater international governance. Mutads mutandis, anintegrated strategy for the enhancement of EC should, in principle, accomplish for theinternational public sector wlbat EC can arguably do for domestic public sectors

The major conclusions that emerge from the analysis are:

(1) The strengthening af evaluation capacity in developingcountries is the mast powerful instrument for reform andincreased efficiency in public sectors.

(2) There is a close relationship between public sectorreform -through evaluation capacity enhancement - and greaterpolitical accountability and improved governance.

(3) The major determinant of the environment which enables thestrengthening of evaluation capacity in developing countries isthe degree to which the markets, information and competitionare allowed to play a role.

(4) The World Bank, multilateral institutions and the donorcommunity, in general, can play a major role in supportingevaluation capacity development in DOs.

(5) Evaluation capacity development should not be envisaged justas an isulated domestic concern of DA. It is a majorinternational issue with global, economic and politicalimplications that need to be addressed with a comprehensivelong-term strategy.

In briet, through its contribution to the correction of "government failure," evaluationhas a "public sector-downscaling and an efficiency-inducing effect" which offers great potentialfor countries searching for sustainable growth and development.

INTRODUCIION

'Economists should cease proffering policy adviceas if they were employed by a benevolent despot,and they should look to the structure within whichpolitical decisions are made.'

-James M. Buchanmn1987

For many developing countries, one of the most striking legacies of the 1980s has beenthe discovery that public sector reform appeasn to be the critical link between macroeconomicorection and sustainable growdth In effect, the experience of the past decade indicates thatwhile m onomic stability is a necessary condition for growth, it alone does not ensure it,and that public sector rest, uauring may well be the key factor in the attainment of durablegrowth. his poses a new and formidable challenge both for developing countries and for themultilateral institutions that wsh to support these reforrs After all, such public sector reforminvolves voluntary1 public and political choices to change, inter abFa, the role of the government,and to modify the existing balance of power and interests.

This paper wil argue that strengthening evaluation capacity (EC) in both developingcountries and multilateral developmental institutions is the best answer to such a challenge.Ihis argument is based on the premise that the ultimate source of financing is efficiency.Without a modicum of efficiency, most sources of financing eventually dry up. This is valid forthe financing of all sorts of projects, policies and programs, be they economic, social, political oreven of an altruistic nature. Without a minimum of efficiency, resources will cease to flow todeveloing countries, to multilateral institutions and to the recipients of aid. This is because theavailability of resourcs ultumately results from public or private decisions based on thesupposition that a given objective is being met-or will be met-with an acceptable degree ofefficiency.

f this premise is accepted, then the question for developing countnes (DCs),multilateral institutions and donor govenments is how to increase resource efficiency! Mhis is a

1

2

difficult question to answer because of the intricate way in which the resources mix,complement, and substitute for one another. In the process, efficiency is often lost andaccountability dissipates. A major part of the World Bank's answer to this question is toimprove the performance of the country portfolio and to 'use resources as effectively aspossible" (Preston 1992). Tbis could be achieved by enhancing the role of its OperationsEvaluation Department (OED), and by refocusing expost evaluation on sustainabledevelopment impact (Wapenhans 1992).

This paper contends that strengthening the evaluation capacity of developing countrieswill lead to increased efficiency, and will constitute, at the same time, a very powerfulinstrument for public sector reform. In brief, strengthened evaluation capacity will bring abouthigher public sector efficiency which is the basic objective of public sector reform.

Summarizing, the main purpose of this paper is to develop the conceptual and analyticalframework that links together: (1) the strengthening of evaluation capacity in developingcountries, (2) public sector reform in those countries, and (3) the role of multilateraldevelopment institutions in support of these efforts. Understanding such a framework isnecessary for the formulation of a consistent strategy that takes into consideration the complexnterrelationships and interests invved and joins them in a cooperative and participatory

effort.3

One final clarification may be in order. In linking evaluation capacity with public sectorreform, this paper assigns particular significance to the evaluation of national policies, incontrast to the exclusive evaluation of projects or programs Evaluation of national policies isemphasized because of their long-term effect on developing countries (Easterly 1992), andbecause reform will not occur without a public sector restructuring that includes changes in themodalities of governmental intervention. Such new strategies for public sector reform will notbe fully forthcoming without the information that emerges from evaluation. The need forpublic sector reform in developing countries is, in reality and primarily, the need to improvepublic sector efficiency. In this sense, public sector reform is tantamount to the correction of"government failure."

L. EVALUAIION CAPACITY IN DEVELOPING COUNTRIES

This chapter deals with the purposes, functions and constraints of evaluation capacity indeveloping countries. It begins with a general definition of evaluation and its purposes, andthen proceeds to analyze the particular situations of most developing countries. Anappreciation for the possible differences between evaluation in industrial countries and inmultilateral institutions, on the one hand, and evaluation in DCs, on the other, wil contributeto the understanding of their respective perspectives and thus to a better design of acoordinated strategy.

a. The Meaning and Objectives of Evaluation

Generally spealing, the term evaluation refers to the systematic and planned collecting,analyzing and reporting of information that can be used to verify compliance with original goalsand to help in making judgments and decisions about future policies, programs or projects(Akdn 1990).

like most definitions of complex concepts, this one is susceptible to various versions,qualifications and caveats. Some experts will say that evaluation is "a process in which, at acertain point in time, the result of a defined output into a system is systematically assessed andreported" (Stokke 1990). Others wil say that 'evaluation is a type of policy research, designedto help people make wise choices about future programming' (Weiss 1990). Yet others wildefine evaluation as a procedure at the interface between fact-finding and value judgements,scientific inquiry and administrative decision-making, a learning process and a justificationexercise" (Musto 1991). Fmally, some authors will emphasize the instrumental character ofevaluation as opposed to social, political or juridical evaluations (Derlien 1990). In brief, thereis no one correct definition of evaluation.

Definitions vary according to institutional setting, methodological bias and the particularpreferences, approaches and objecdves of the evaluators. It would seem, however, that few

3

4

experts would disagree with the notion that 'evaluation is an evolving work of art, a process inwhich accumulating experience in steadily allowing the boundaries oG analysis to be pushed

forward and the policy conclusions to become increasingly substantiale (Cassen 1986).

In view of the above, it is clear that evaluation is characterized, at least, by the following

three core elements and/or concepts: (1) it is a proces., (2) it must be systematic and planned,

and (3) it must be based on data previously established.

If definitions fail to express the meaning-or all of the meanings-of evaluation, a

consideration of its objectives and purposes may contribute to a better understanding of the

concept.

The main purpose of evaluation is to assess results against original expectations and to

learn from experience. Evaluation attempts to verify that anticipated results indeed occurred

and to determine why they did or did not occur as planned. This establishes the accountability

for the results obtained. The theory behind this practice is that if information regarding

performance is fed back into the process, it will contnbute to better design and implementation

of projects, programs and policies. Within this context, the main objective of performance

evaluation is to contribute to the efficient attainment of pre-established goals

The dynamics of evaluation will normally lead not only to the verification of the

attainment of pre-established objectives, but also to an examination of the intrinsic merit or

desirability of the objectives themselves. A rigorous evaluation will also scrutinize the

conditions under which it is carried out to assure the validity of its conclusions. In this respect,

evaluation is analogous to an experiment in which a hypothesis is put to test. The evaluator

should take care to differentiate between the hypothesis and the experiment.4

b. Requements ad Constants

Performance evaluation is part of a cycle that begis with an appraisal or an er ante

evaluation, continues with an implementation phase-in which budget allocations are made and

financial and managerial monitoring takes place-and ends with the original goals being

contrasted with the results obtained, and the information derried then being fed back into

programming In this cycle, many requirements must be satisfied, but the feedback is the sine

qua non condition (Stokke 1991). After all, evaluation would be wasted if its findings were not

5

used. Evaluation does not end with reporting. It could almost be said that evaluation beginswith reporting.

This feedback requirement raises two issues- first, how does one ensure that it actuallytakes place and, second and more relevantly, how does one ensure that the information isincorporated? To achieve the first objective, it would be necessary-although not sufficient-tohave close proximity between the evaluators and the object of the evaluation. To achieve thesecond objective, a clear separation between the evaluators and the evaluated would be advised,thereby lending credence to the results obtaind. If the results of the evaluation are suspectand lack credibility, they will not be fed back IThere is then the possibility of conflict between(1) the requirement of proximity and familiarity, to achieve relevance and feedback, and (2) therequirement of independence and objectivity to achieve credibility.

Therc does not seem to exist one single solution to reconcile these two requirements.The particulars of each case, institution, sector or country will gradually determine whatarrangement wil work. According to P. Viveret (1989), "What matters is not the location of thefunction but the guarantee of its independence." Whether or not the results are disseminated,and feedback induces real changes in policies and programs, will be the test of evaluation(Wholey, Newcomer and Associates 1989). If no changes are perceived by public sectormanagers and by the public at large, the system will not be crecdble, and although nominalevaluations may continue, they will accomplish little.

C. The Objetives of Evalhaon Capacity In Developing Countries

When considering evaluation in developing countries, the basic question is whether thegeneral considerations stated above relating to the meaning, objectives and constraints ofevaluation are valid and applicable, and if not, what are the differences and how should they bedealt with in order for these countries to strengthen their evaluation capacity.

It could be said that the basic tenets of almost all definitions of evaluation would beapplicable to the case of developing countries. That is, it does not seem necesary to develop aparticular definiton for these countries. This does not mean, however, that the assumptionsimplicit in most general definitions would normally apply to many developing countries. Therequirement that evaluation be planned, systematic, and databased, for example, wil not be metin many cases But this does not warrant a redefinition of evaluation or the search for a new,

6

ad hoc definition. It is important to bear in mind the particular conditions and restrictions that

characterize different groups of developing countries.

It would be difficult to argue that the objectives of EC in developing countries are much

different from those in industrial countries. And yet it seems that in DCs, certain dimensions of

these objectives acquire particular relevance. While the general objectives may be the same for

both groups of countries, i.e., to assess results against criginal expectations and to learn from

experience, it could not be said they would necessarily be the main route to public sector reform

or increased political accountability in developed countries. There is also a difference in the

degree to which evaluation is expected to modify the modality of government intervention in

one group of countries vis-a-vis the other.5 Fmally, it could hardly be said that evaluation

capacity has the same significance in developed countries in terms of its contribution to better

governance that it has in developing countries. In brief, while the objectives of evaluation may

have an abstract or universal validity in principle, such objectives and purposes may differ in

practice depending on the particular conditions 3f the developing countries being considered.

The objectives of evaluation capacity in developing countries can generally be classified

on a global or macroeconomic level, and a program or project leveL On a global or

macroeconomic level, the main purpose is to raise the quality of public investment and the long-

term productivity of the economy, and to improve the effectiveness of public policies. The

expectation is that if senior public policy makers receive information on the results of their

policies, they will be able to better assess the developmental impact of those policies. On a

program or project level, evaluation capacity seeks to achieve better management of public

programs and of specific investment projects, as well as establish managerial accountability from

public sector managers and bureaucrats.

On the macroeconomic level, it is important to distinguish between political decisions

about sectoral priorities, and the global policy and institutional framework in which those

governmental programs are expected to be implemented. This second sublevel has great

importance because it determines whether the specific projects or programs are carried out

within a propitious, neutral or adverse environment. That is, it determines whether (1) the

market forces are granted an important role, (2) external evaluations are accepted, or (3)

guarantees are provided by the national governmenL

There are interdependencies between these levels. When results of specific projects or

program evaluations are fed back 'upwards,' senior policy makers are better able to appreciate

7

the effectiveness of their policies. Also, if policies and the development strategy from 'the top'are market-oriented rather than interventionist, it will be less difficult-and necessaly-to carryout evaluations "at the bottom." According to C Rist (1990), this is "an interactive,retrospective and prospective" process between program evaluations and policy analysis.

Macroeconomic policies can also be evaluated through a 'counter factual scenario' inwhich possible outcomes are assumed from alternative policy frameworks. While this approachis useful, its actual implementation is difficult as the constraints are very limiting and thehypothetical courses of action are open to many questions. For this reason, the evaluation ofmacroeconomic policy "from the bottom," with concrete results, seems a preferable altemative.Generafly, evaluation of projects or programs will be less difficult than evaluation of globalmacroeconomic or sectoral policies.

IL EVALUATION CAPACTY, PUBUC SECTOR REFORM AND GOVERNANCE INDEVELOPING COUNTRIES

The ultimate and principal purpose of evaluation in developing countries is to improvepublic sector efficiency. At the same time, the main objective of public sector reform is toimprove public sector performance and to provide a propitious institutional enovironment frgrowth and development. This interdependence offers the opportunity to design a strategy thatwill accomplish the two objecives. The premnise of this paper is that the best strategy for publicsector reform in developing countries is to strengthen evaluation capacity. This does not meanthat all other efforts to reform public sectors, Le, institutional building, the fostering ofefficiency through competition, privatization and decentralized management, should beabandoned. It means, nevertheless, that since resources are scarce-financial, human andnstitutional-it would be more effective to concentrate on building up evaluation capacity than

to maintain a dispersed effort across the numerous problematic areas of public sectos

a. Public Sector Reform

Basically, there are three main arguments in favor of prioritizing evaluation capacity asthe primary instrument for public sector reform.

(1) The development and strengthening of evaluation capacity makes thecorrection of 'government failure more manageable and politicallyviable.

(2) The processes to achieve economic goals and political objcivesreinforce each other.

(3) The institutionalization of evaluation and its incorporation into thepolitical process strengthens governance.

8

9

Th.e Pubi, Secoxrdmswmadng a.d , udu g Effect of E IPot Evalualo

Ae bblk Sec_g~~~ ~~~~~ d::*i Eaw i:_-ds is resk o* Fdic. iects anDCs ten b-c wayia,gc-Din nmnl tea andqit inefficien anwa

tefms, that is ui trms of what c j l To a lae e-den is ai e rl o apvi 1fepW men Sro on as a rerome t a J*tfail, butto afipdly dear with p=ug soda!ad Mpola pvbkemL Lide; if any', atnio is paid to the.paiiuar modai and daeiics of- uch inwdkn4 r which often sinply lead to publcsector n and to is -m" who take advantage: of the situadton at hand

Th pefomnc evalao "o - and of hid'-raoncom itndpnjeu lad: to a better appreciaon -o the -nhen limtatos of ublic wice?-and of the

possibiis4offered by nmd-orinted I i of intrventon TW a propitous ewonment iscreawed to downscale the size-or t r-duoce the rate of gwt-of pubic sectors and to induceefficiey 2 ad beAefa rpnc .-

One of the main reans why it iS so difficult to carry out public sector reform is

because it is seldom dictated by the market In contrast, external adjustment and, to someexet, maoeconomic adjustment are ultimately disztad by market forces. When thispveremptory adjustment' takes place, policy makers encounter less resistance because the publicsees the reforms as inevitable. Other reforms, Le., improving public sector performance, aresedom if ever dictated by the marlet Policy makers advocating these ypes of reforms have topush them through the political process in order for them to finally be enacted through law.Tis gies ample room and time for the creation of what H. Hughes (1985) cals 'politicalobstacles. This can be defined as the exrcise of political power by 'vested interests' (Mueller1986), that is, those groups who gain from large, inefflcient, interventionist public sectors. Veryoften, these interests, which are normaly better organized than the public at large, wil be ableto impede or restrict the scope of public sector reforms (Olson 1982). In other words, internaladjustment is, to some extent, avoidable-or at least postponable-while external adjustment isinevitable (Wiesner D. 19&88)

Evaluation capacity can achive public sector reform because it is less conspicuous thanglobal public sector reform schemes. It can begin with specific studies carried out byindependent evaluators, and the results can be gradually revealed, inter alia, to the public.Reform-minded policy makers and some poltical leaders may use these results to gain politicalinfluence. Although the process would be slow, resisted, and not always work, it hastremendous potential to make efficiency politically profitable. Surely the power of tvested

10

interest wvill not disappear, and rent-seekers will continue to exs&t (Buchanan, Tollison andTullock 1980). However, evaluation has the capacity to change the political discourse about the

role of the state and the modalities of intervention. It can de-ideologize such discourse andmake it more pragmatic. Its main contribution would be to gradually create a constituency foreffective government. This would be decisive in terms of the political viability of public sector

reforms. After all, 'political issues are much more difficult and important than technical issues"(Alesina 1992). If political support exsts, reform will be viable.

Evaluation not only increases the likelihood of greater efficiency, but simultaneouslystrengthens the political process through which public choices are made. As public participationgrows-awakened by the results of evaluation-public choices will gain legitimacy, and those

holding power will become more politically accountable.

Box2Pebffic Sector Ref D t by Maet Forces or Negotafted througlh tf. Molltical

O fi main reasom why i. E so, d.'cu to c- - pl c .- is beca,se

-t-:s seldom diated Jy market omc oi a advocaiw pubi sector h tpus itef s t ghocp w tansp firim a lconstiueng T-higmes -amp k--room- ad r t v e gsto epal obswacle an toivv~pede or-restr-ct-the scope ~f public sector tefom.- As a resut the publc rare has a chanceto seea ub -c s rwc iar a- d conions. STh efficency that woudresul iomsch- sw<ko -i *thep4be reali~the- pulc- obsees whe thei miae

-*~~~~~~~~~~~~~~~hi -th narketostimlate extrnalajWtnwn ThMe.lin"h deteloping countrie can gr~ on fordeas.with a- minimum of external balantz but eOp gly ii t q public sedo.:

In brief, the correction of "government failure' involves policies that limit the extent ofinterventions, which in tum determine the size of the public sector and the conditions which

induce efficiency in that public sector. Evaluation capacity has a relative advantage over other

instruments of reform because it better reveals the effectiveness of development models and

specific policies and programs, as well as the limitations of particular modalites of intervention.

This does not mean, however, that decentralization of decision making, divestiture, orinstitutional building are not adequate tool- for public sector reform, but that evaluation has the

relative advantage.

11

: - ~~~~~~~~Bo 3

-Te Effectof PoltIcAINVgilance On Eonomic and Socia Efficency

Evaluatbn seems to be tdh prefemble ztwnent for publc sector reform because it notonly iscreases the likelhood of greater economic and social efflckncy, but simultaneousb(-strengthens the polialprocess throp which public choices are made. Political developmentand gains in efficency reinforce each odier. Just as 'mai et failure has been poicized(Buchanan 1991)6 to Justf the proleftson of inaventions, it would seem that governmentfaiwre has not been suffciety poiticied to create a consiuncy for evauation and publicsector reform. As publc pankicpao grows-awakened by the dseman of evauationresus -publc choices l in pi- t , and rtose holding power wilf be more politicallyaccountabler Politcalvinci has a posie ffect on onomic efficeny.

Fmally, it should be said that evaluation is not the only instrument of accountability.

There are others of an institutional and structural character that play an important role, e.g., thevigilance exercised by the legislative branch of government, elections, and tnbunals of accounts.

However, in many DCs, these instruments often play only a nominal role in accountability in the

sense that their results and information seldom lead to significant changes in policies or to

greater political openness.

b Governance and Evaluation Capacity

The third argument in support of strengthening evaluation is that it contributes

significantly to better govemance. The term govemance has become a sort of new paradigm

blessed with numerous virtues but burdened with multiple meanings. It seems to have

originated in the search for the underlying causes of underdevelopment, and the realization that

external and macroeconomic adjustment are not enough to sustain growth and to bring about

political and social development. While the term itself is a neutral concept meaning simply the

'exercise of authority and control or, more broadly, government" (Brautigam 1991), it is often

t3ken as a synonym for a democratic government which allows the economic and politicalmarkets to function freely in the pursuit of the colective welfare. This type of government

leads to-and results from-greater public participation, more political accountability, efficient

interventions and sustainable economic growth it is no wonder, therefore, that governance, ie.,

sound development management, is perceived as a panacea by policy reformers, developmental

iltitutions, donor countries and aid agencies.

12

.. .. ..... . 'i0 - -. f.-. . . .--~~~~~~~~~~~~ ... . . - ' ...0-'6*

-~~~~~~~~~~~~~~... . ...-.. .. B*tii fC 4C d

EyauatoaVlsI-is the Istraets fPubli S.ctor Reaom

many de'kp* oads aecm oteralrtg htpb secw tyor fm a ther

cmInto of# a number of pBoram and -baie,L, ingsitkml uidin&- civi vc:rfonn nd dcnia DC havec-baked othsand -aming ridnna the ight

- ' ^. ''' cMdxiu c

are npu ad t of evaluaio h ie n b e ta o orm on evaut

ma m and moeac tabili i: a idwkedt-le fho n hi with specfgc

n metinwhichrfo t induced anedasa wh m- ae re f

. 6iw--.::::.. .. .. These0 ;t f

The relationship between gonance and development is omplex and inconclusive

There are examples of countries where economic growth has been remarkable, but no

substantial political development has taken place and whlere governance is tantamount to order

and the exercise of authority and power. InI other cases, governance and the execs ofauthority, while springing from incipient democracy, have not been accompanied ty durblegrowth or significant development. What seems less intractable, nevertheless, is the relationshipbetween public sector performance and development It appears that without a minimum of

public setor performance, little if any growth can take place.

This paper defines governance as the result of the combined economic and political

pronssestch rou wcthe public reveals its preferences, and finances them through the fiscaland economic sstem (Wiesner D. 1991). Strengthening EC will bring about improved

governance. Te infomnation that results from EC and its eflcts on the public's perception of

the consequences of governmental intervention will serve as a springboard for improvedgovernance. Since public sectors are the realm of gonmental intehrention, the development

of EC wil also transform public sector management. In tis framerk, EC acts both as a

supply of inputs for better governance and as a dreand for hbiger economic efficiency and

political accountability.

13

Govemance is both a result and an input of evaluation. Without a modicum of

governance and a propitious political climate for economic markets to function freely, it is

unlikely that evaluation capacity will thrive and reform public sectors. At the same time, thedevelopment of EC leads to public accountability and the emergence of a constituency for

public sector reform which is critical for improved governance. Through evaluation,

governments end up receiving increasingly legitimate mandates which allow them to exercise

power and authority while remaining politically accountable for their actions. Evaluation will

eventually be institutionalized and become part of the everyday activity of governments.

Institutionalized evaluation within the government, and in the political processes that underlie

all forms of government, will perform a task similar to that of the market forces in the privatesector. That is, it will assure that efficiency is the test of sustainability.

Box 5Performance Evaluatiom A "Field of Dreams' or a Plausible Instrument for Public Sector

Reform?

This inot an easy quesion to answer, andperhaps evaluation should not be cornerd intoproving tat it is apanacea. After all; nothing can prove to be apanacea ora sufficent condiionon an ec ante basis withou becoming a tautology. Er post evaluation provides a good answer tothe queson of what the strategy for public sector reform should be It can hard be that alltoo- , intuments or approaches for reform are equally effecfive. Are cvil service reform, overall-instional buildn decentralization and evaluation itself equaffy effective in refo1mng pubfficsecton in DCs? Should budgets and multilateml support for public sector reforn be equaflyallocated among the numerous approaches to public sector reform? Or should a strategy besought to determite what insuments should be given relative prioityW

a post evaluaton has a relatve advantage over other instunments of refonm because it: (1)-links rmsut orperfonnunce more directly with budget allocations and policy deczsions, and thwsoffers an eplecit icnve for public sector effiiency; (2) works not on on the economic aspectsof developmenr but also on the polical and governance dimensions; and (3) is more effectve inrevealing "govemmtent faiure" or the flaws of partular interventionist develop l strategiesThis. particulaly me when, in addition to verifying the attabiment of a set of objectves,evaluaion leads to an amminaton of the validity or approprateness of the objectives themselves.

In shot, in relation to otier means of public sector refonm, evaluation seems more iketo serve as the public sector manager's surrogate for the profit-and-loss statement of the pnvatesector. In this respect as R Piciotto (1993) has put it, "evalation is to the public sector whataccounting is to the private sector."

14

As can be seen, there is a strong political dimension to evaluation, particularly whenviewed more as an instrument of public sector reform than only as a means to increase

efficiency of individual institutions. It would be valid, therefore, to question the assumption,

implicit in the previous paragraphs, that evaluation would be accepted as a panacea and as a

new paradigm with little political resistance. Certainly this will not be the case, and it should be

recognized that evaluation and "evaluators as change agents" (Sonnichsen 1989) face an

extremely difficult task. After all, evaluation implies:

An uneasy relationship between cognitive and normative statements, scientificcapabilities and administrative requirements, objective, knowledge andsubjective convictions or interesL....Evaluation is a highly sensitive proceduresubject to the forces exerted by divergent, conflicting interests located at theinterface between science and politics, sincerity and consideration, factualvalidity and normative responsibility (Musto 1991).

Perhaps it can be said that 'in evaluation, feedback follows relevance...and politics." It is

quite possible to have a situation in which 'politics" control the process of evaluation and

restrict its capacity to contribute to governance. However, if evaluation is strengthened and

given a fair chance, it may well prove to be pivotal in bringing about governance and public

sector reform.

To sum up, evaluation is here to stay. It is a rapidly growing field of academic and

political interest in all countries and institutions, public and private, creditor and debtor.Evaluation is the interest of all and in the interest of alL

Box 6Publk Sector Reform through Evaluation: Specilc Country Experences

While it may sdtl be too early to report on empirical evidence of the success or failure ofpublic sector reform dutugh evabuaton, it can be said that a number of countnes are beginning tograsp tze full potental of this iurmoent In several of zhese counties, iLe, Morocco, Brazi4Colombia; Ch*na, Chik and Venezuea, authorities have gained speciwl awareness of thestrengths-and exigencies-of evaluation as a key instrument of public sector refonn, and are-begqinig to establish programs to inplement thi strategy.

The World Banl, through its Evaluation Capabilit Development Prorm (ECDn) andother channels, is providing support for these efforts. T7teir joint endeavors wil render lessons toDCs and multatral instudon both of which have a special stake in the purit of eq .ency.

111. THE POLITICAL ECONOMY OF EVALUATION CAPACITY IN DEVELOPING

COUNTRIES

In the preceding two sections, the logic and rationale were offered for making

evaluation capacity the backbone of public sector reform in developing countries. The analysis

led to clear, normative conclusions and recommendations, i.e., DCs should adopt new and

better policies. However, it would be naive to believe that all that was wanting was a rational

policy nostrum. This would assume 'a benign and welfare-maximizing state...whose purpose is to

do good" (Grindle 1989). From this perspective, failure to achieve "good" results is largely

explained by incomplete information and faulty analysis.

The harsh realities of developing countries and the inherent limitations of "public

choices"7 indicate that, instead of assuming a "welfare-maximizing state," it should be assumed

that strong political opposition will emerge from those who would lose from the reforms. This

is the realm of the "new political economy,' which explains the existence of irrational collective

policy and the inability or unwillingness of many governments to respond to the "public interest."

a. Guiding Principles for Reform

If offering a "logical" policy recommendation is not enough for it to become a law or

common practice, what then can be done? There are no simple answers to this question. R.

Inman (1987) suggests that "to alter performance one must alter the structure of government,"

and recommends that economic policy analysis continue to search for institutional reforms that

facilitate the adoption of "good"' policies. The more basic answer would be to exercise vigilance

over and limit the activities of government in order to minimize the profits of interest groups

from intervention.

More specifically, a strategy for public sector reform and evaluation capacity

development in DCs should be built around the following four guiding principles.'0

15

16

(1) Allow for market forces or surrogates to play a role, particularly in the

financing of the intervention and its cvaluation.

(2) Build er ante into the intervention itself a mechanism for independent

or self-evaluation.

(3) Encourage competition from outside and within the government, and

link budget allocations and managcment incentives to institutional

performance.

(4) Establish mechanisms for the independent collection of data on public

sector activity and for the dissemination of evaluation results.

In applying the first guiding principle, all proposals for governmental intervention should

question both the source of the proposal and its financing. This is the fundamental starting

point of all efforts to reduce inefficient interventions and to induce efficient interventions.

The nature and source of financing is the principal determinant of efficiency. If

financing comes from the budget, guarantees, ear-marked funds, prices or tariffs where there is

a public monopoly, or from external sources (aid or loans) where the accountability link is weak

and diffuse, it should be presumed that efficiency is in jeopardy. When financing comes from

the markets or from market-like surrogates, and there is risk involved, it should be presumed

that the project, program or policy in question has a high chance of being reasonably efficient.

In brief, the source of financing determines the quality of the expenditure. A soft budget

constraint (Shirley and Nellis 1991) militates against the development of an EC systenL

However, the introduction of this paper stated that efficiency is the only sustainable

source of financing Is there not a contradiction here? Not really. Under a market

perspective, resources flow in response to efficiency and productivity prospects. Under a public

sector perspective, markets play a minimum role, and through "public choices," resources are

allocated even when efficiency is lacking.

The application of the second guiding principle should follow A. Wildavsky's (1992)

dictum that "the ideal organization should be self-evaluating." If self-evaluation is not feasible,

ecternal evaluations could be built, er ante, into the design of a particular intervention. One

way of ensuring external evaluation is to stipulate in the legislation for new programs or

17

institutions that they not be renewed or maintained until an independent evaluation is carried

ouL

The third element in the strategy to create the conditions for efficient intervention and

successful evaluation calls for a search for ways in which governmental and nongovemmental

entities can participate. There are several ways in which this could work. One possibility is to

take a "hard budget position' in which demonstrated efficiency is a condition of budget

allocations (Israel 1987)." This demonstration may take the form of independent or self-

evaluation, or simply of survival when programs face market-like conditions and competition.

Box 7The Reform and Modernization of Colombia's Public Sector through Ex Post Evaluation

Colombia's 1991 constitution mandated the design of a system to evaluate investnentprojects andpolicies on the national; state and local levels. The NationalPlanningDeparlmen4in charge of allocating the national investment budget, has strengthened its mechanisms forbrnging evaluation results into the budgeting process and sectoral policy formulatin The publcsector at huae has begun to instutionalize evaluation. While not abandoning other efforts toimprove public sector perfonmance, the authorities are sponsoring kgislation that widZ, in effec4render ex post evaluation the major instbiment of public sector reformL

Under die aegis of the National Plannung Department and in collabortion with theWorld Bank, an international semizar was held in Santa Marta in November 1991 to rvew thefindig of five case studies, and to discuss the possble design of a national sytm of evahuton.Beyond iividual ex post project evaluafton, the basic idea was to Ore-examine the quaty ofpublic invesmnt and of publc exendiues in general" (Montenegro 1992).

This enhanced role for evaluation is not an isolated efforn but rather underscores amajor change in Colombia's developmental strategy. Beginning in 1990, a process of trade andfinancial liberalization was initiated an:4 on the whole, market forces were given a greater role inprivate and public sector resource allocation. This is thepropitious environment that evaluationneeds to develop its fid potentiaL

In many developing countries, natural monopolies for public services have been largely

responsible for inefficiencies and "non-performing public sectors." Privatization is often not

feasible for political reasons. In such cases, it may be possible to encourage competition by

having different public providers of public services (Elazar 1991). Competition within the

government will allow consumers of public services or goods to choose between providers, or, in

18

S. Paul's (1991) terms, to voice their choices and exit from where there is no micro

accountability.

Paul makes a useful distinction between (1) 'macro-upward accountability," (2) "micro

level accountability," and (3) "outward-horizontal accountability." The first refers to the highest

aggregate accountability, ie., macroeconomic policy, and to measures that determine the overalllevel of output and employment The second refers to the accountability that pertains to

specific public services. The third focuses on the "outward," or horizontal, accountability

towards the public in general. The inefficiencies of many public activities often result from thelack of communication and accountability between these levels, and because the expanding role

of the state overloads the system, particularly at the macro leveL The public, on the whole,does not become a real constituency to demand accountability on each level because it is

misinformed and lacks sufficient understanding of the interrelationships between macro andmicro policies (Moore 1992).

The fourth and last guiding principle advocates the independent collection of

information and data as a means of improving public policies. Many observers have noted that

the common lack of information in DCs is not necessarily fortuitous (Wiesner D. 1992). Veryoften, "bad" policies and inefficient institutions have been self-perpetuated because theinformation available "justifies" them. Those who gain from "bad' policies often control or are

the only source of the information. The "losers do not always know how much they are losing

and the winners incur costs to hide information from them" (Birdsall and James 1990). In this

context, information becomes policy. By the same token, new or independent information could

lead to new and better policies. This reinforces the need for research to support what might be

called better public decision maldng (Chelirnsky 1991).

b. Normative Recommendations and Positive Realities

The application of these guidelines will not eliminate inefficiencies altogether, but will

limit the extent to which interventions cause inefficiencies. Furthermore, a strategy based on

these parameters will elicit "good" politics in the sense that competition and information will

bring about new coalitions that mave a direct interest in sustaining growth-enhancing reforms"

(Lamb and Weaving 1992).

Normative recommendations are not enough to induce public sector reform and provide

EC with a propitious institutional and policy environment. While the analytical process and

19

empirical research are necessary to determine what may be the preferred technical solution,

they are not sufficient to transform public sectors in DCs. Little is gained by saying, forinstance, that public institutions should be austere and efficient, and that their managers should

be honest and competenL These exhortations will bring less of the desired results than astrategy based on changing the real conditions (Spinoza 16 7 7)U and the policy environment in

which those institutions and managers perform. This means that frugality, efficiency andcompetence will occur when budget transfers become conditioned, when extemal competition

forces restructuring, and when public sector pay and seniority are linked to performance. In

brief, markets are important, but they are not enough (Eggertsson 19 90 ).'

The preceding paragraphs lead to the conclusion that EC could be the instrument for

public sector reform. However, the question remains as to how the decision to enhance EC

would come about when, in principle, public choices do not automatically respond to the publicinterest. This is the classical dilemma of public choice theory, with no simple or unique

solution. The process may take many directions, but they are not totally random. Basically,

efforts need to be focused on the general conditions, assuming that they are the main

determinants of outcomes. In developing countries, these conditions may result from localpolitical leadership, 'demonstration effects," information, or the dynamics of change that are

engendered by abysmally incompetent public sectors. All of these factors are conducive to

'better" public choices.

The second way to circumvent deleterious public choices is to pose the policy issues in

terms of global, structural and constitutional reforms that appear remote and "behind a veil"

which disguises some of their real implications. Giving independence to central banks,

eliminating ear-marked funds, or balancing the budget may be included in these kinds of

reforms. In Colombia, for instance, a 1991 constitutional reform called for the establishment of

a system of evaluation in the public sector. With this mandate, Colombia stands well-poised to

develop a national evaluation system.

IV. THE INSlTIUTIONAL FRAMEWORK FOR EVALUATION CAPACITY

DEVELOPMENT IN DEVELOPING COUNTRIES

The purpose of this chapter is to delineate a strategy for the institutionalization of EC

in developing countries. It begins with an analysis of the factors that constitute the first level of

an enabling environment. It then offers some considerations about the critical importance of

the strategic placement of the institution encharged with EC development. In the third section,

alternative approaches for launching, implementing and consolidating EC systems are offered.

In the fourth and last section, some brief comments are offered about the role of the private

sector.

a. The Enabling Environment for Evaluation Capacity Development

There are several conditions needed for launching and implementing EC systems. These

conditions vary from country to country depending, principally, on the degree to which markets

are allowed to determine the allocation of public and private resources. This, in turn, governs

the options available to support evaluation in developing countries.

The conditions that constitute an environment conducive to the development of EC

include:

(1) public demand that policies and priorities be determined by an open

political exchange;

(2) political demand that public funds be used more efficiently;

(3) a macroeconomic model and a development strategy that are market-

oriented; and

20

21

(4) an open political system in which information can flow freely and bringabout greater political accountability and improved gvernance.

These factors promote performance evaluation by stimulating the demand for evaluationand by increasing the supply of information from evaluations. It is difficult to rank these fatorsaccording to relative importance. Their effectiveness in bringing about EC in developing

countries will be determined by tbeir interplay, and individual diagnostics of the countrysituation will indicate which requirement is most needed and what is the proper sequnene orreform.

b. The Itituallon ef Eludato. Capadty

Once a govemment has decided to launch an EC program, it will need to determinewhere to begin and what steps to take. Although the approach will depend on the strength ofthe existing institutional environment, the strategic placement of the main evaluation office is ofcritical importace (Chelmsky 1985). Chapter I indicated that the feedback of evaluationresults is the sine que non condition for the implementation of EC To assure that thiscondition is met, it is advisable to locate EC where planing and budget decisio are madeMiis enhances the probability that the results wil be fed back, although it also poses the risk ofjeopardiing the credibility of the results. Notwithstanding this constraint, it seems that the best

place to begin would be the Mlnistry of Fmance or the Miniy of Plannin& It is importantthat, within the orational structure, evaluation be institutionalized 'from the top down"(Wholey, Newcomer and Associates 1989).

The extension of EC will basically depend on how proficient the public sector is, andhow quiwkly it can adopt EC as a normal tool of public administration. The objective sbould beto develop a system of evaluation with several planning and budget linkages. Althougb theproces would best begin where major policy decisions are made, special efforts should be madeto propagate evaluation to other levels of goernment.

The m_chansms and processes of extending evaluation from one main and central unitto a national system will undoubtedly vary from country to country, and will be affected by manycircuutancest but credibility will be imperative for EC to become a reality. The final test ofEC is its credibilty to the public, to public managers, to politicians, to the private sector and tolabor unions. In essence, the credibility of the government to reform the public sector and toadopt a political and economic model of development based on efficiency and the market will

22

depend on the credibility of EC. Such credibility, by itself, would be an expression of improved

governance (Moore 1993). It would mean that those managers, institutions and programs that

are able to argue that they are efficient, in effect, get a larger share of budgets and resources

(Wiesner D. 1991). When this happens, the system will become self-sustaining.

C. Alternative Approaches and Restnctions

An examination of the possible approaches to launching, implementing and consolidating

an evaluation system reveals few specific models that can be followed. There are multiple

alternative paths to EC development, and each depends on factors that vary considerably. 'The

national development of evaluation depends on a number of factors and constellations" (Derlien

1990). Whether it should be a centralized or decentralized system, concentrate on macro or

micro aspects, or start at the executive or legislative branches will depend on the particular

politicaL institutional and human resource conditions of each country.

A government choosing to launch an evaluation system could enact a legal mandate with

some general or specific instructions for EC implementation. This mandate would send an

important signal to the public at large, to the public sector managers and to the legislativebranch of the goverment. It would convey the message that performance will have a major

bearing on resource allocation (Jones 1991). The credibility of this stand will depend on the

effective implementation of the policy.

Many restrictions and difficulties will initially arise from: (1) insufficient information, (2)

management and staffing limitations, and (3) divergent or inconsistent policies across the

spectrum of government. In the initial stages of development of an evaluation system in DCs,

proper policy coordination will be a major challenge (Lamb 1990). It will not be easy to have a

coherent policy when so many aspects of governmental intervention permit only a partial

implementation of an EC effort. To deal with this situation, a centralized policy authority may

be the best answer.

Another possi'ble way to launch an EC system may be to conduct specific external

evaluations of particular investment projects, programs or policies. Through these case studies,

considerable learning could take place about what may be worling, why, or why not. While this

approach seems attractive, it also faces difficult constraints. Who will decide what to evaluate?

For whom? What methodologies should be used? How can inappropriate conclusions or the

"bad" politiciation of these initial efforts be prevented?

23

When choosing models for the initiation of an EC system, another question that arises iswhether the effort should begin by evaluating macroeconomic policies, or specific micro projectsor programs. From a general perspecfive, it would be better to start by examining the rationaleof the macro aspects of policies. After all, most programs and projects originate from amacroeconomic policy perspective of the economy.

In short, the constraints of the possible approaches result from what may be called thedisabling environment. This environment is characterzed by an interventionist economic andpolitical modeL a weak public sector institutional framework, a closed political system, and avery limited human resource base. These factors could comprise the criteria for theconstruction of a typology for groups of countries."4

d. The Role of the Private Sector

In developing countries, the private sector should offer political support to agovermnental effort to develop an evaluation system, and should understand that, in the longrun, a successful EC system will be in its interest. More specifically, in many countries theprivate sector should "privatize itselL' By doing so, it would contnbute to an environmentwherein markets would have a larger role in public sector management and in the allocation ofpublic and private resources. This 'privatization of the private sector' is a very important factorbecause in many DCs, there is often a comfortable partnership between the public sector andthe private sector where inefficient and unmarket-like interventions-with little if any externalevaluations-are welcomed by both sectors. The private sector becomes a partner in theintervention and supports it politically. The public sector is subject to little criticism because, asM. Moore (1993) has pointed out, the public typically lacks information and power for aneffective appraisal and sanctioning function.

V. THE STRATEGY FOR EVALUATION CAPACrIY DEVELOPMENT IN

DEVELOPING COUNTRIES AND THE ROLE OF MULTILATERAL INSTITUTIONS

A strategy for the development of evaluation capacity in DCs and for the role ofmultilateral institutions should begin with three global considerations. Frst, there is a "two-wayrelationship" in that external lending and aid affect evaluation capacity in DCs, and the

development of EC in these countries in tumn influences: (1) the level and characteristics of

lending by multilateral institutions, and (2) the 'in-house" evaluations of operations. A second

consideration is that multilateral lending is often perceived as a special type of 'public good' bysome DCs and even by some of the providers of such resources. This affects the EC

relationship between DCs and multilateral institutions. A third global fiator that needs to beconsidered is that just as there is a "national political economy," there is also an "internationalpolitical economy." This is the institutional environment in which lending takes place, aid isdonated, and debtor and creditor countries try to evaluate what they all are doing. Thisinternational political economy environment conditions the relationships between the differentparties and, in particular, their efforts to be more efficient and accountable.

S. Mulidlatral Fnace and Ald: A Public Good?

Chapter m examined the relationship between the source of financing and the efficiencywith which resources are used. Tbis leads to the question of the conditions under whichmultilateral lending and aid wil be conducive to the development of EC in DCs

The answer depends, to a large extent, on how DCs perceive such external transfer ofresources. f such financing is largey perceived as a special type of "public good," the thesis ofthis paper remains that neither efficiency nor local evaluation capacity wil benefit substantially.That is, if the financing does not come mostly from capital markets, if there is little riskassociated with it, and !here is a "soft-budgetW position behind it, it is not likely that the use ofsuch resources will be efficient and that much effort will be made by recipients to evaluate itsperformance.

24

25

On the contrary, if those resource transfers represent real full costs, recipient countrieswill be more vigilant about their use, and evaluation efforts will be more substantial and no lessintent than for their own resources. In answer to the initial question, if creditors and aidrecipients perceive multilateral lending and aid to be largely a 'public good,' such resourcetransfer will not, in most cases, be conducive to-nor encourage-the development of rigorous ECin DCs. Resource transfer will most certainly not encourage the strict evaluation of theperformance of those particular loans and forms of aid. This wil have a rather adverse effecton the totality of DCs' efforts to improvr ex post evaluation in general.

Ex -=:--o st Ealuation anidEnergy SedCtr Reormin hi: Colombia1~~~~~~~~~fd ::.: :E;:- - : .. ". . : ---:=..

Colombia's ene,~, sector serves asamdd how expost eva'ao a lead to pubfic

1 :i970-1987 (World B-a-I4peYvtons_ a (OED), Repo No. 8893,: Jwu1 :990),: the -. omii.. :got' etiniiae a mmstwig of its. enegyap ectw; mupran

,ess:.. .... 99^

lesn lere a: build: -. --:::g me-- :n- .s for pefean e-lato int th ne p .olicy:: .:, . -

Foll g a 11 - .: smiatri- S Mar Coombia, th s of.pa 'oiivnq~ctcmea th ovmmnpipar Iegig*ztio allwin farprvatesor

a nC bn an Who secel a f rie for f so . It aLo e t1 : ........ .. ,. .: ........................... , :f::, . :. e.... ......-------...... to.................. foumlton of Watioala EnaDjSaueg> .ir cotast tope)c setoaplanhg It IS d~ffclto7 bnaginet aow eerj etor fon takinig pe withouti te rpost evauaio condcte

fy- : evalaionskyabo offe: a'good:t'amle of the we of qp ilinsitiosa etob gonestha wish to rengthen thr eva ion stymL In

'thispatiua h an' e........... cas.5 Banks ORD .. . . I. - --, . : --- -: ::-: '.: .S,,Sy -: ,., .- ,:~~~. . . -. : , :.. :. g '.- :,. .,S

When -examining the perspectives of the debtor countries and the recipients of aid, it isimportant to distinguish between the two levels of perception about the 3public goodness oflending and of aid. On a macroeconomic level, and in the case of multilateral lending, there ismost like little perception that such finance is a "public good.7 Most Ministers of Fnance andHeads of Planning Institutions know that there is always a cost However, in other areas orlevels of govements, the notion often exists that such financing is vituall free. When theseconflicting perceptions are not adequately resolved, resources are used ineffciently and

26

evaluation capacity in DCs is not helped. In the case of aid, the local perception of 'publicgood' is more pervasive, and the consequences for efficiency and for EC are more severe.

be Intenational Political Economy and Governae

If multilateral resources are not strictly seen as 'public goods," then what explains thelack of higher developmental impact and improved evaluation capacity? While there are manyanswers to this question, one of the main explanations is that just as the domestic sphereexperiences are all kinds of difficulties with governmental intervention, public choices andpolitical constraints, so do international economic and political institutions, perhaps even moreso. After all, how and to whom are international and multilateral economic institutionsaccountable? How can accountability be ascertained when principal-agent' relationships' areso ambiguous and labyrinthine? How are international public choices made? Who are the free-riders? Is there any logic in international collective action? And, finally, where is internationalgovernance? As J. E. Spero (1984) puts it:

In the twentieth century the study of international political economy has beenneglected. Politics and economics have been divorced from each other andisolated in analysis and theory-Consequently, international political economyhas been fragmented into international politics and international economics.

These considerations shift the analysis into the field of international political economy,and the application of public choice theory into international economics. This offers "freshinsights and a unified theory of human behavior as well as a technical apparatus" (Frey 1991) tosearch for solutions.

The specific insight that this paper wishes to offer is that evaluation capacitydevelopment by DCs, by international and multilateral public institutions, as weDl as by aidagencies,' could be the framework to bring about higher efficiency and accountability to alL Itwould also contribute to the formulation of more consistent international policies and to greaterinternational governance. Mutads mutandis, an integrated strategy for the enhancement of ECshould, in principle, accomplish for the international public sector what it has been argued thatEC can do for domestic public sectors.

While this paper cannot examine all of the possible analogies and externalities that exstbetween domestic public sectors and the international public sector, it submits the folowingthree propositions:

27

(1) To strengthen evaluation capacity in DCs, a global perspective of the

problem and an integrated international strategy are indispensable.

(2) While the strategy should be a communal effort, it should begin

basically as a priority of DCs.

(3) Multilateral lending institutions can play a crucial role in

complementing DCs' efforts to strengthen their evaluation capacity.

Without a global and integrated strategy, it would be difficult for any party to really

make substantial progress in the key areas of interesL There is thus a need for a macro

enabling environment that will support efforts by: (1) DCs to strengthen evaluation capacity,

and (2) multilateral institutions to improve portfolio performance, enhance their own evaluation

capacity infrastructure, and support EC development programs by DC-.

All these activities are interrelated and there are close interdependencies among them.'If, for instance, DCs did not effectively assign a high priority to their EC, how far could lending

institutions progress in strengthening their own EC and in improving the quality of entry-or

exit-of their projects? All parties have a stake in each other's respective "upstreami and

'downstream" phases of loans and projects. In short, a coordinated and a global strategy is the

key requirement for a macro-institutional enabling environment.

C. A Multilateral Strategy in Support of Developing Countries' Efforts

Both DCs and multilateral institutions have a stake in the design of an EC strategy. It is

in the interest of both parties to help each other strengthen their respective ECs. I developing

cour,tries had well-structured evaluation capacities, there would not be much additional need for

evaluation enhancement within the multilateral institutions. Their country portfolios would

perform satisfactorily and they would seldom be asked-or allowed-to lend for projects with low

prospects of success. If, on the other hand, multilateral institutions had a more stringent-and

independent-appraisal process they would seldom end up providing loans with poor prospects.

The following actions are recommended for the Bank to take in support of DCs' efforts

to strengthen their evaluation capacity.

28

(1) Ijnk loans and tcbcnical cooperation for public sector reform toevaluation capacity devlopment in DCs.

(2) Build into the appraisal phase of project the a ante conditions toficilitate evaluation -downstreame through impact evaluation.

(3) Build EC development into the incentive structure of Bankmanagement

(4) Strengthen the Evaluation Capability Development Program (ECDP).

(5) Support resarch and disemination of information.

(6) Provide training, btuf alia, through the Economic DevelopmentInstitute.

(7) Provide technical assistance and support "horizontal exchanges amongDCA

(8) Provide financing through the Institutional Development Fund (IDF),or a similar fund, to support research and specific programs of ECsengtening,

Evaluation support to DCs has to be adopted by the Bank 3s a m;ar and long-termistitutional cmmitment If the Bank comes to the conclusion, as this paper has, that EC is themast adequae insrument for public sector reform in DCs, it should translate this into policyad implement it, inter aha, through bans, technical assistance, tang and1 programs such asECP.

The strateg that the Bank adopts should be flezble in order to take into considerationthe mrad of posaible county conditions. The implementation should proced by supportingthe develmnt of indigenous networks of organizations and people responsible for key poicypocsems' (Lamb 1987). That is, the initiatine has to come fiom the borrowig countries. Theyhae to own the programs.

29

From the start, the Bank should be fully conscious of the magnitude of the task and ofthe resistances it will face. It will be criticized and misinterpreted; it will have many of the samediffiwlties that an EC initiative is greeted with in DCA In addition, results wil be slow incoming. Evaluation capacity development in DCO is a long-term endeavor.

VI. CONCLUDING REMARKS

This paper has tried to provide the conceptual and analytical framework to examine-in

an integrated way-three interdependent processes, namely: (1) the strengthening of evaluation

capacity in developing countries; (2) public sector reform in those countries; and (3) the role

that multilateral institutions and the World Bank, in particular, may play in support of the

efforts of DCs. An understanding of the inner relationships between these processes is critical

to a comprehensive approach to addressing the numerous issues involved.

The conclusions of this paper are summarized as follows:

(1) Strengthening evaluation capacity in developing countries is the most

powerful instrument for achieving public sector reform and efficiency.

(2) There is a close relationship between public sector refonn -through

evaluation capacity enhancement-and the achievement of greater

political accountability and improved govt ance

(3) The major determinant of the enabling environment for the

strengthening of evaluation capacity in developing countries is the

degree to which markets and competition are allowed to play a role.

(4) Evaluation capacity development should not be envisaged just as an

isolated domestic concern of developing countries. It is a major

international issue with global, economic and political implications that

need to be addressed with a comprehensive long-term strategy.

(5) The World Bank, the multilateral instit ions and the donor

community, in generaL can play a major role in supporting evaluationcapacity development in DQ.

30

31

The recommended use of evaluation capacity as the main instument to reform publicsectors in DCs goes beyond the scope of a sectoral improvement In effect, this paper hasargued that developing countries wishing to moa from macreonomic and external balancecorrection to faster and more sustainable growth should find in EC building the route to moreefficient public sectors, and thus to a more productive economy as a whole. While public sectorreform in these countries requires many actions across several fionts, EC seems to be thegalvanizng mechanism to bring together dispersed efforts and to concentrate them in onecohesive endeavor. This does not mean that all other efforts to reform public sectors, iLeinstitutional building, the fostering of efficiency through competition, and decentralizedmanagement, should be abandoned. It means, nevertheless, that since re 3urces are scarce, itwould be more effective to concentrate on building up EC than to maintain a dispersed effortacross the numerous problematic areas in public sectos

The strengthening of evaluation capacity in developing countres makes the correction of'government failure' more viable and brings transparency to the political processes throughwhich public choices are made. Evaluation has the capacity to change political discourse aboutthe role of the state and the modalities of governmental nterentions. It can 'de-ideologize'such discourse and make it more pragmatic. One of its major contributions is that it wilgradually create a constituency for effective government Political development and gains ineconomic efficiency reiforce each other. As the public participates more, awakened byevaluation results, public choices wfll become more legitimate, and there wil be greater politicalaccountability and improved govemanc.

However, these changes will not happen easily, quickly and without resistance fromthose who stand to lose from reformL Nor will change occur simply because there is anormative recmmendation to 'have better policies.' It would be naive to believe that all thatwas wanting was a rational policy nostrum. would assume a benip and welfar-maximizingstate whose pmpose was to do good. The harsh realities of DCs and the inherent limitations ofpublic choices indicate that public sector reform, induced by evaluation capacity development,will come about gradually, and only after an enormous effort by the developing countries and bythe international commuiity.

Just as the domestic sphere expiences are all kinds of difficulties with governmentalintervention, public choices and politcal constraints, so do international economic and politicalinstitutions, perhaps even more so. After all, how and by whom are intemational public choicesmade? How is the accountabil of internatonal and multilateral institutions determined?

32

Who is the 'principal' and who is the 'agent?" Who are the "free-ridersW in the interplaybetween DCs, developed countries, national and international bureaucracies? Is there any logicin international collective action? And, finally, where is international governance? In bnef, theglobal counterpart of the 'national political economy' is the 'intemational political economy'environment which conditions the relationships between the different parties and, in particular,their efforts to be more efficient and accountable.

The conceptual framework of this paper springs from two fundamental and interrelatedpremises, namely that: (1) the ultimate source of financing is efficiency, and (2) the source offinancing is the principal determinant of efficiency.

These two apparently contradictory postulates underline the interrelationships betweenthe two perspectives or dimensions from which problems of public policy-domestic andinternational-should be analyzed:

(1) the market exchange perspective, where resources autonomously flow in pursuitof-and are attracted by-efficiency, and

(2) the political economy, or public choice perspective that implies that it ispossible to obtain financing that is not supported by efficiency.

When trying to reform pvblic sectors and to strengthen EC, the source of financing shouldalways be questioned. Virtually eveything starts and ends with the answer to this question.

A strategy for EC building in developing countries cannot be formulated in isolation ofevaluation activities in developed countries and in multilateral and aid agencies. This is a 'two-way relationship' in which multilateral lending, technical assistance and aid affect evaluationcapacity in DCs but, at the same time, the development of EC in these countries influences: (1)the level and characteristics of resource transfers, and (2) the 'in-house' evaluations of creditorinstitutions and aid agencies

These activities are interrelated and there are close interdependencies among them. IfDCs did not really assign high priority to their evaluation capacity infrastructure, how far couldlending or aid institutions go in supporting those efforts and in strengthening their own EC?How far could they go in improving the quality of entry-or exit-of their projects and programs?If DCs had a well-structured EC, there would not be much additional need for evaluation

33

enhancement within the multilateral and aid institutions. Their country portfolios would tend toperform satisfactorily and they would seldom be asked-or aDlowed-to lend for projects with poorprospects of success. If the 'principal-agent' relationships between countries and themultilateral institutions were not so ambiguous and labyrinthine, there would be greateraccountability and efficiency, and EC activities would be less difficulL In brief, all parties have astake in what each one is doing in its respective "upstream" or 'downstream' phases of loans orprograms. A coordinated global strategy is the key policy requirement for successfuldevelopment of EC by al parties.

In addition to a global strategy, there is a specific role for the World Bank to supportDCs' efforts to strengthen evaluation capacity. The first step should be to prioritize theenhancement of EC in its lending and technical assistance programs. In addition, there couldbe a number of concrete actions such as: (1) building up its Evaluation Capability DevelopmentProgram; (2) financing research, seminars and publications; (3) providing specialized trainingthrough the Economic Development Institute; (4) expanding the Institutional DevelopmentFund to support specific program by DCs; and (5) building EC development into the incentivesuacture of dhe Bank's senior management. Beyond these specific activities, the Bank wouldneed to adopt a long-term commitment to support DCs wishing to enhance their evaluationcapacity.

One concluding caveat may be in order. The question may be raised whether this paperhas not overstated the case for evaluation, and has not assumed conditions in DCs, which, ifpresent, would mean that they were in fact developed countries. This question revealsparticular insight into the complex dynamics of reform in DCs. Indeed, one can seldom identifyer ante all of the pre-conditions or elements that mix to determine a given outcome.Nevertheless, this paper contends that not all of the pre-conditions indicated above need to bemet for er post evaluation to play a central role. It further emphasizes the crucial link betweenevaluation results, and the formulation of policies and budget allocations. This linkage need notcover the whole spectrum of the public sector. In fact, it would be impossible to do so. Whatis necessary is to establish this linkage in a number of selected policy and project areas. Fromthat point on, the process will have a reasonable chance of success, and evaluation wfll graduallyreform public sectors.

NOTES

1. To a large extent, external adjustment and a minimum of macroeconomic balance endup being established by the market forces independently of deliberate policy measures.This is a sort of de facto adjustment that takes place outside the legal or establishedinstitutional framework Policy measures to remedy other kinds of structural rigiditiesand inefficiencies have to be explicit in nature, go through a transparent political processand be expressed in specific legislation. Policy makers proposing legislative reforms willfind strong resistance as they will be perceived as making voluntary choices, whereasopponents will see proposed reforms not as inevitable, but rather as discretionarychoices. In contrast to legislative reforms, adjustments that come impersonally and aredictated by the market will be less resisted because they will be seen as inevitable.Public sector reform is so difficult precisely because it is so seldom dictated by themarket. On this subject see Wiesner D., Eduardo, (1988), "Latin America's PolicyResponse to the Debt Crisis: Learning from Adversity,' Internaional EconomicCooperation, ed. Martin Feldstein, Chicago: The University of Chicago Press, p. 29&

2. It is no coincidence that some of the best technical work in the area of evaluation hasbeen carried out by large aid donors. Tlhis technical work and the growing interest inthis field is largely explained by the size of aid provided by these countries and theirdesire to assure an efficient use. In 1989, Official Development Assistance amounted toUS$513 billion, and 90 percent of this aid was provided by the members of the OECD'sDevelopment Assistance Committee. It is understandable then that there is aconstituency for evaluation in these countries, unlilke in developing countries. OECD,(1990), Development Cooperation: Efforts and Policies of dhe Members of theDevelopment Assistance Comm;ttee, Report, Paris.

3. The terms "participation" and "participatory development" have come to highlight theimportance of involvement in decision-making processes. Participation has also beenviewed "as a vital complement to macroeconomic adjustment" (Picciotto 1992) and aspart of institutional development The term is used here to underline the possibleimplications of participation in organizational design and institutional cooperation. Theidea is that efforts to strengthen evaluation should be carried out in close coordinationand collaboration by the multilateral institutions and developing countries. This doesnot mean, however, that evaluation capacity building should not be pursued in its ownright by developing countries.

35

36

4. This is sometimes referred to as the validation problem- in which reality or knowledgeis only that which, according to KI Popper, resists refutation. Tlis is the same approachfofowed in Milton Friedman's theory of positivism in which hypotheses cannot bevaidated; they can only fail to be refuted. For a rigorous analysis of the complenties ofcausal relationships and the theory of evaluation, see Sechrest, L B., and A. G. Scott(1993), "Understading Causes and Generaliing about Them," New Dbwgons forIvgvnm Evaluadon, The American Evaluation Association, No. 57, Spring, 1993.

S. Geneally, it would be accepted that governmental intervention in DCs allows littleroom for market forces. In contrast, intervention in deveoped countries is often a toolfor the protection of the "invisible hand," and albows market forces to affect outcomes.

6. On the politicization of market failure and its implications for efficiency, see Buchaan,J, (1991), The Economics and the Edis of Co u Order, Ann Arbor. TheUnversity of Michgan Prs

7. In 1981, Professor A Bates asked, "Why should reasonable men adopt public policicsthat have harmfil consequences for the society they govern?" The public choiceliterature developed by J. Buchanan (1962), W. Niskanen (1971), A. Downs (1967) andG. Tullock (1965) would answer by saying: because those men are not motivated by thepublic welfare, but by their particular individual interests. Politicians, bureaucrats, laborleader, et aL are for the most part defending their particular interests when makingdeciions or espousmg public policies. They are "rentseekers pursuing themaximization of their own welfare and not that of the public. There is also the immense

difficulty of defining what the "public interest" is, and how and by whom it is revealed.In bref; public choices are suspect, and to overcome their limitations policy maSersshould try to establish codffons that are conducive to the desired results. 'Decontant search for mechanisms that bring the market forces into govermentinterventions is one of these conditions. For an excellent discussion of the process thatleads to a good proxy of the "public interest" or of the social welfare function, seeBuchanan, J., and G. Tullock, (1962), T7he Cakulis of Consent, Chapter 19, Ann Arbor:The Univrsity of Michigan Press.

8 Fcw terms have the strong tradition and ability to constantly renovate themselves as do:"political economy," "new political economy," "positive political economy' and"nol political economy." Surely they cannot aD mean the same thing or betotally different. iterature on these concepts is vast, varied and growing. Theunderlying theme is the classic question (Inman 1997) of "under what circumstances aremarimes or governnts the preferred institution for allocating societal resourc?" lhiquestion cannot be resolved only by analtcal considerations. That is, once analyticalreasonig has led to the conclusion that government intervention is preferable, suchintervention may fail. The basic conclusion is that just as markets are imperfect, so tooare gvemnts.

9. Itis wt easy to define 'good and "bad policies. It is ever more difficdt to define"good" and "bad politics." However, in both cases, "badness" and "goodnessW are rlative

37

to whether the policies or politics contnbute to the liberalization of economic and/orpolitical processes. The underlying premise is that economic and political openness leadto efficiency and -etter governance.

10. In addition to these guiding principles for reform, an aecellent set of specific principlesfor evaluation development can be found in the OECD's 1992 Development AssistanceCommittee Manual, Principles for Evaluation of Development Assistance.'

11. For a superb discussion of the budgeting process acting as the ultimate competitionsurrogate, see Israel, Arturo, (1987). Israel states, "In theory, there is no reason why thebudget should not play such a role, but most countries would have to make majormodificalions in their current practices if they were to put the budget to thi use."

12. Much of the effort to improve public administration in developing countries is oftenspent on diatribes against indolence, nepotism, dishonesty and incompetence bymanagers and civil servants. In 1677, Benedict Spinoza observed in his TractatwPoliie that it should be assumed that men are motivated principally by considerationsof interest. He took human behavior as an empirical fact and set out "to constuctpolitical institutions in such a fashion that individuals, acting in pursuit of their owninterests, will be led, by the institutional structure within which such action takes place,to further the interests of their fellow members. See abo Buchanan, J, and G. Tullock,(1962), lhe Caluhus of Consent, Ann Arbor. Ihe University of Michigan Press, p. 313.

13. In addition to the pioneering work of Douglas C. North on the importanmc ofsinsitutional factors in explaining differences in country economic performance, Thrain

Eggertsson's 1990 Economic Behavwor and Instuons, Cambridge University Press, is anoutstanding new piece In it he develops the tenets of what he calls 'NeoinstitutionalEoonomics (NIE), which, while not rejecting neoclassical economics, warns thatautomatic, unrestricted market exchange is not alvays assumable and that, therefore,odter factors, Le., institutional, are the determinants of economic and performanceoutcomes.

14. The issue of the relationship between conditions in developing countries-theirtypologies-and public choices leading to refonns and an enabling enviroment for EC:has paricular relevance for the design of a strategy for the World Bank to support theserefrms. If adequate typologies could be consucted and if the conditions encapsulatedin each category had some predictive powers about the receptiveness for ECdevelopment, the Bankes response could be more easily structured and implmentedHowever, it does not seem that an attempt to construct typologics would go vey far. Atbest, broad categories addressing the followang components would need to be broughtin:

(1) The degree to which the markets play a role in public and private resourcealocation.

(2) The proficiency of public sector institutions.(3)i The quality of the human resource base.

38

(4) The degee of political openness and the extent to which the government isaccountable to its citizens.

These elements could be used to construct a proxy for country receptiveness to reformsand to EC partnerships for the Bank. However, the approach contemplated in theBank's ECDP -starting out from each individual country which expresses an interest indeveloping its own EC-appears most sensible. The premise behind this approach is thatthere are a myriad of possible country conditions and situations. Under thesecircumstances, the three phases leading to a detailed diagnosis and to the design of anevaluation framework seem realistic.

15. "Principal-agente theory examines the costs that arise when the "principal" engages an"agent' to perform some service which involves delegating some decision makingauthority to the "agent." Both parties are assumed to be utlity-maxmizing, but there isno a pron reason for their respective utility functions to coincide. It is possible,therefore, that the "agent'se actions may not agree with the best interests of the principal(Heymann 1988). The whole point of the principal-agent relationship is that there arebenefits from the specaliation that results when different parties perform differentrokls In the particular relationship between multflateral banks and debtor countries, itis not totally clear who is the "principal" and who is the "agent." A more difficult issue ishow to reconcile possible divergences in interest between the parties. mhis whole areaof "principal-agent' relationships between multilateral institutions and borrowingcountries is a vast and new field of tenra icognita waiting to be explored.

16. In a recent work, T. Killick (1990) analyzes the effectiveness of aid in Africa and findsthat, taking the region as a whole, aid received by Sub-Saharan countries has beenineffective in developmental terms. Among the factors he mentions that limit sucheffetiveness are the proliferation of projects, diverse policy conditionality, insufficientdonor coordination and conflicting political interests. While these problems are verycomplcz, one way to address them would be by strengthening evaluation capacity inrecipient countries. Curiously, this is not one of the solutions contemplated by ProfessorKillick when searching for ways to enhance aid effectiveness. See Killick, Tony, 'TheDevelopmental Effectiveness of AID to Africa," World Bank, Working Paper No. 646,October 1990, p. 43.

17. For a discussion on the interdependencies between different EC paths among countries,see Derlien, Hans-Ulrich, (1990), "Genesis and Structure of Evaluation Efforts inComparative Perspecive, Pgram Evaouton and the Management of Gornment, ed.C Ray Rist, Transaction Publishes

BIBUOGRAPHCAL REFERENCES

1. Alesina, A., (1992), Political Models of Macroeconomic Policy and Fiscal Reform, WorldBank Working Paper No. 970, Washington, D.C

2. Alkin, C Marvin, (1990), Debates on Evauaton, Part H, London: SAGE Publications Inc.,p. 81.

3. Asian Development Bank, (1992), whe Enabling Enviromnent for Evaluation," Summaryof Proceedings of the Regional Seminar on Performance Evaluation in Asia and the Pacific,Appendix 4, p. 165.

4. Bates, A., (1981), Towad a Politcal Economy of Development A Radonal ChoicePersectve, Berkeley: University of Califomnia Press.

5. Birdsall, Nancy and E. James, (1990), EJffcincy and Equity in Social Spending: How andWhy Goverments Misbehave, World Bank Working Paper No. 274, Washington, D.C., p.31.

6. Brautigam, D., (1991), Goence and Economy, World Bank Working Paper No. 815,Washington, D.C.

7. Buchanan, J., (1991), The Economics and the Ethics of Constitutonal Order, Ann ArborThe Univeity of Michigan Press.

& , and G. Tulock, (1962), The Calclu of Consent, Ann Arbor: The University ofMichigan Press.

9. R. Tollison, and G. Tullock, (1980), Toward a Theoly of the Rent-Seeking Society,Tcxas A&M Univesity Press.

10. Cassen, R, (1986), Does Aid Worc? Oxford: Clarendon Press.

11. Chelimsky, E, ed., (1985), "Old Patterns and New Directions in Program Evaluation,"Pro'uwnm Evaluation Pattens andDirecions, Washington, D.C.: American Society for PublicAdministration.

12. , (1991), "On the Social Science Contnbution to Governmental Decision-Making"Science, VoL 254, Oct.

39

40

13. Derlien, Hans-Ulrich, (1990), 'Program Evaluation in the Federal Republic of Germany,"F)ogrnm Evaluaion and the Management of Govemment, ed. C Ray Rist, TransactionPublishers, pp. 37 and 147.

14. Downs, A., (1967), Inside Bureaucracy, Boston: Little Brown.

15. Easterly, William, et al, (1992), How do National Poicies Affect Long-run Growth? WorldBank Discussion Paper No. 164, Washington, D.C., p. 39.

16. Eggertsson, Thrainn, (1990), Economic Behavior and Instiutons, Cambridge UniversityPress.

17. Elazar, Daniel, (1991), 'Cooperative Federalism," Competition Among States and LocalGovenments, ed. D. Kenyon and J. Kinkaid, Washington, D.C.: The Urban Institute Press,p. 65.

18 Frey, Bruno, (1991), 'The Public Choice View of International Political Economy," ThePoiical Economy of Intemational Oaniations, ed. R. Vaubel and T. Willet, WestviewPress, p. 20.

19. Grindle, Merilee, (1989), The New Political Economy: Positive Economics and NegativePo-liics, World Bank Working Paper No. 304, Washington, D.C, p. 6.

20. Heymann, David, (1988), Input Controls in the Public Sector: Rhat Does Economic TheoryOffer? IMF Working Paper 88159, Washington, D.C., July.

21. Hughes, Helen, (1985), 'Policy Lessons of the Development Experience," OECD, NewYorkc Group of Thirty.

22. Tnman, Robert, (1987), "Markets, Governments and the New Political Economy," Handbookof Pubic Economics, VoL II, ed. Auerbach and Feldstein, North Holland, p. 649.

23. Israet Artuo, (1987), Institonl DevelopmenL Incentnves to Performance, Baltimore,London: Johns Hopkins University Press, p. 126.

24. Joint Inspection Unit, JIU, (1982), United Nations System Cooperation in DevelopingEvaluatin by Govenmun, ed. Alred Forde and Earl Sohm, Geneva.

25. Jones, Leroy, (1991), Peformance Evalationfor Public Ernteprises, World Bank DiscusionPaper No. 122, Washington, D.C.

26. Killick, Tony, (1990), The Devdopmenta Effeteness of AID to Aflica, World BankWorking Paper No. 646, Washington, D.C., October, p. 43.

27. Lamb, Geoffrey, (1987),ManagingEconomcPolicyChange InsionalDimensions, WorldBank Discussion Paper No. 14, Washington, D.C, p. 25.

41

28. , and Rachel Weaving, (1992), Managing Policy Refonr in tdh Real World, WorldBank, EDI Seminar Series, Washington, D.C.

29. Montenegro, Armando (1992), Contoly Evaluacid6n de la Gesd6n Pabaca, DNP, Bogota,Colombia, p. 10.

30. Moore, Mick, (1993), -Declining to Learn from the East: The World Bank on Governanceand Development,' IDS Bufletin, Vol. 24, No. 1.

31. Musto, Stefan, (1991), 'Evaluation Policy and Performance in the Fedenl Republic ofGermany," Evaluaing Development Assistance: Policies and Perfomuance, ed. Olav Stodke,London: Frank Cass, pp. 127-148.

32. Niskanen, W., (1971), Bureaucnocy and Representative Govenment, Chicago: AldineAtherton.

33. OECD, (1990), Development Cooperaon. Effois and Policies of the Membem of theDevelopment Assisnce Committee, Report, Paris.

34. OECD, (1992), 'Principles for Evaluation of Development Assistance,' DevelpmentAssistance ManuaL DAC Pincwes for Effective Aid.

35. Olson, M., (1982), The Rise and Decline of Nations, New Haven: Yale University Press.

36. Paul, Samuel, (1991), Strengthening Public Senice Accouttability, World Bank DiscussionPaper No. 136, Washington, D.C, p. 12.

37. Picciotto, Robert, (1992), Participation Develpment Myths and Demas, World BankWorlkng Paper No. 930, July.

38 , (1993), 'Visibility and Disappointment: The Evohing Role of DevelopmentEvaluation.' Publication forthcoming, World Bank, Washington, D.C.

39. Preston, Lewis, (1992), 'Address to the Board of Governors,' Proccedings from World BankBoard of Govemors Meeting, World Bank, Washington, D.C, Sept 22, p. 11.

40. Rist, C. Ray, ed., (1990), "Managing Evaluations or Managing by Evaluations: Choices andConsequences,' Prognrm Evaluaton and the Management of Govenment Pattems andProspect aoss Eigt Nations, New Brunsw.ick Transaction Publishers.

41. Shirley, Mary, and John Nellis, (1991), Publc Enterpise Refomv T7e Lessons ofEpeknce,World Bank, EDI Development Studies, Washington, D.C., p. 9.

42. Sonnichsen, R. C, (1989), 'roducing Evaluations that Make an Impact,' ImpvvingGovernmentPerfomance Evaluation StategisforBuidbgig k Perfwoance Programs, ed.Joseph Wholcy and K. E. Newcomer, San Francisco: lossey-Bass Publishers.

42

43. Spero, J. E., (1977), The Poliidcs of Intemational Economic Rdations, London: Allen andUnwin.

44. Spinoza, Benedict, (1677),A Treatise on Politics, trans William Maccafll London: Holjoakl1854.

45. Stokke, Olav, (1990), "Policies, Performance, Trends and Challenges in Aid Evaluation,"Policies and Perfonnance, ecL Frank Cass, London: Frank Cass, p. 3.

46. Viveret, Patrick, (1989), L'Evaluation des Poitques et des Acdons Pubques, Paris: LaDocumentation Francaise, p. 101.

47. Weiss, Carol, (1990), Debates on Evaluation, ed. Marvin AUdn, London: SAGEPublications, Inc., p. 83.

4& Wholey, Joseph, Newcomer, K E. and Associates, (1989), Improving GovernmentPefomance: Evaluation Stratgies for Buiding Hgh Perfornance Programs, San Francisco:Jossey-Bass Publishers, p. 195.

49. Wiesner D., Eduardo, (1991), Colombia: Descentralizaci6n y Federalismo Fiscal, InfornieFmal de la Misidn para la Descentralizacid6n y las Fmanzas, Presidencia de la Republica yDepartamento Nacional de Planeaci6n, Bogoti, Colombia, p. 373.

50. . (1992), "El Disefio de un Sistema de Evaluaci6n de Resultados: Principios yOpciones,g Contol y Evaluacidn de la Geudn Pubfica, Departamento Nacional dePlaneaci6n, Bogotfi, Colombia, p. 123.

51. - , (1988) "Latin America's Policy Response to the Debt Crisis: Learning fromAdversity," Intemational Econom Cooperaion, ed. Marn Feldstein, 'chicago: TheUniversity of Chicago Press, p. 298.

52. -, (1992) Politico, Energetca y EsJctv Insaionl en Colombia, Comisi6nNacional de Energia, Mmisterio de Minas y Energia, Bogotfi, Colombia.

53. Wildavsky, Aaron, (1972), The Self-EvaluatingOrganization,"PubicAdminiumonReview.Reprinted in Evaluation Studies ReviewAnnual, Vol. 3.

54. World Bank, (1991), The Challenge of Development," World Develpment Report 1991,New Yorkc Oxford University Press, Chapter 7.

55. , (1991), "Managing Technical Assistane in the 1990s," Board of Directors Paper,Committee on Cost Effectiveness and Budget Practices, Washington, D.C, Nov.

43

This paper also draws on infonnation from the following documents which are not available to thepublic.

Mueller, Dennis, (1986), "The Growth of Goverment: A Public Choice Perspectiv," IMF,Departmental Memorandum, Washington, D.C.

World Bankc, (1992), "Effective Implementation: Key to Development Impact," PortfolioManagement Task Force, Washington, D.C., Nov.

Distributors of World Bank PublicationsAIWEIInA 1 Miii. fMaObinw KENYA SOUTH ANM §CA. DO*VI4Co"M. SIM, 4L*uIft AME&Su*.UAJL W. friMwGdoftC_ Ca Qi Home,I_ h _dum"flftm

Abaftin k-nmk Cape Tom. UAUUTILAU&IA.PAUNSWGUUIIA PAX kms 11 KORMR. KENJEC OFKJLuOOmmW lAND, 41101 HdhI 10 ft. Km.. .. cmp.mum rmw""M&Nft&U_ FRA No P£._1

VANUATU, ANDWTN SAOA PAt km Ill. Kw 1 bie .. umm 1 nmSlD.AkbJmma SEAC S.. a km410m4WIM_h _ Pa Wail kaikPilmll O.~.MIm31S UiLauedmU_ MALAlIA &a 24Vidal 7511Pb UMI,iMimkymcaqmindw

vbuhhap,Lhiih SPAINAUSTRA GcUANY rim am 1W_. MEN Pu gm M..i.p,..m%S.A.CaM ad Ca. WJOVmg SWil KOWA Lupw C_MDl3Cb.b 31 FIlIES lUlAlMmdii£1011 WI.. 00.Sim ACO

RMS U l _1 AScUANLAOUM HONGKOCMhtACAO Apmabi.t.Iai Cornl soCom_ JmiuIn Am i mIa ID lIMlpm.bdMDJ. oHn

AmdfbMll- AS14543 WoiihmqH s Ia 14 W_bic.ia NEDm_LANO R LANKAANDTHE MLI)MRSObmimWMRIAM 2ii kd HP.e De UadS/Mb1JWdM OIH_fI _ai

0db 1 CmIruI Hoag Kg RO kon2 PA onl G7410A EH_mlbm I11.rChbamiMmA.

154.NwA _i dS AHMlbiUdb hi isLiL NSWZKALAND CAdmbaCaMnq"4 7D Mt Raid ECONELW

mamd-6m4Mo Prwal Uqs" SWWUN7&.OJLAv Now Mm F lionKiwaNI R_^ orm Aud P_

IS ix I4.widI mug 3qihm12.3.UUGum hmisd Emw NCKA3 SbhumtlJliDaLmey o a oy.M UrbuilyftmuvidAv.do MPal 1kwcuaw.in3idIagjew Fu?p.w10kmd. 13114 AIAJiRomd Pd,mo1mugU ADWII

NewDdli-12f012km .OPMUCANADA S471 r308dmIEua.ur C7_MM*Au NORWAYC.P.9I1fru.A.mp. Calnib-7W072 New.... h>amUinC swNUAMDkmdiuwUIlQibw bwkik Om*u Fwmk1g. DF"aDm Jmimum Hood 3.5dmn PMA 412SEk_wad LbuIih P,mt

5du11_mLCmmd N.0Cab4 Lu.degCICUL kmg.M.- 0Oo C0l4 WlmtbW.iocLA. PAIOSTANA_-mV.p.d.Kab II. 38.U9KKaIVX MI _ Book FAw*

wAagp C~#A.. Raid &.hahaub.Q.i.d Lb 19htyaHydbod-SW1107 PA 3BNo.729 So ImdnA'" m

CHINA szbmSU Cmp'm. U_

rmwbbftNi NwTh b.mg.Nama.am FMV5.DAh>SO.qJl AhindmId-300S EdbIaID.wrASA TANZWAARaft ApmbdmS4 iadui.rdtyFli

CALW 16.AAa^JkMag M.d.a RoadWm_m1nIl. I-Ad- 2241 PHIIUTWFn DarmSI

4 @iAm300 hImbi _ C.EraDBIL Canll zRnEd su"I17SCImd la THAMAND

ElUs"N"w CWmbd=Towwi C.l Dn Stocon vivo= NoMm4*010 AymbA Nra. dILV 30dsim godCmwdlZdKImdd.DWwbm em.zdwi oo.

A _icdmWIM INDONEDIA Makd.Mmpb, M0UP.541 PR. dldmLAmhod TRNIAD&TOMACO0,ANtKAAbidIi06PIAMM lab 9mobodutll POLA?Q NARMDA.MARAADO&

PASwIII _b..dwmPlUbl wS"m DObINICA.CRENADA. GKUINAcKWUS Jaloftuma 1 IlL Rd..a3I/= JAMAW&A. MPlTRAT.SiCaI.A4id R. 004 Wi. ICTRNlVU.ST.LUaLuC~p.aIIqa IYALAND s.vNcwT&CgNADR6.IE,i..Sm. Gsv.m.SupSmgmy Fwhakmi. 5SyawbiiUit

rAaalls sop 1S*R Il Ab sw .fwmuPonk2 4-Hwa.at"t2d IPJ.aN gwdbsw.Nla_ DOth 2 ULOb.3 0fep

ez414fiw _ TfmiWwthdl

JI _ Y._ liwab LU L SORTUCAL TURl3.owAMIK P.OLowmS U,ua1aft-aS InlaiK-O IAduiumgC TdA,i,UM KmDoCa744 NwdiapSoL No.U

WOLhbm C4ph.JuOCaN IEIC ITALY hm_br

S Cim.TAwmC,pA. LU G sCoaiidUuithRdUSA SAUDIARA,AQTAR_e1_SaconeIm"bckmftJw VipD=MCA%b l/K Ju kIM,. WaNIIUNCDOU

Ap.aMido.GC 212041 c.sbPmaIdhSC PAIsm215 limb.b Li._D_womobp SOW FSb. lU. iKi P70B=3

Afmll " GUO2FG5GM,AI MRZAUUCOU JAPAN SRGAFOTNWAN E%ImdAl&h E _ _ftAIJCQMkM H T Chvm.Rmnkp4.Er 113 IdmikP vUinJlcab.O Ti,. RIL. LlIamkddkm

coiammWb. Ap W

.~ ~ ~ ~ ~ ~ ~ ~ ~~~~~~~~~1 .akgtdSCmn.. . -~~~~~~3

Recent World Bank Discussion Papers fintaeui

No. 1 4 .i4ridrlmnral Reeardii In S.tliernti .1i'ria: .4 I:Fu,ua.iaewaLq/r .4roAa. Andircw Spurlmilg. Teck Y. P-,c. (;O,dwi

Mkamianga. and Christopher Nkw;nvJna

Noi. 185 Mialit.ary i:xiiulefdire and Ef lion,mill i)ne1ionpent .4 Synip1.an oil 111 Rtearhla I.arv l-itedd 1v (G;cotirrv I.-nibi withtV.lcrtan.. Kaild,h

ro lI:it E /1i.§J;a .-y .,tyan .Sdi,itiraion 11ij I mll nft) .Alblsremueim 1iilre, ( J1* St.dieit- Ie um AtiticrtiIil nd Willauii (m: Cjvvltimb

No. 187 7Ire Sian' IlAhng.. (.;paenlp : IIIiO l and fl)ptuun Ani all Ktilaur

No. 18 X Indageraoa,naz o/ 1411111 .uuaStlc lar Iiu'zrand'r,udtera SlihltonlII I ). Dvl. ediltor

No. I8') ovlert),'ry. 1q.mlariittti'. and the l:nvnltnimrnt Steplici ). Niniik

No. 19(1 .\ItuIN ai 4 rl ( an 1 ierilopa (: .antran I :i 'ahtill- tile Itn irt tr the 1:raqh- F ,amr,mant Joliih I l1 icr

No1. I' .Al.prollran ire .1 Lt. ron o1nan 1j ./1a, t anl t I lIr,1,t )peu i:... tPa I .ifpliS) d.qhl A hnIted

No19 11r2 l t letit -ank lpIilrlill.j: I;rl i li-Exenttir *a,d Srsarr:.y lljori i Wclcllu%i and otticrs

No. 193 Ilnforananion .Syrtean S%tratraeriNr l,'IFl l iin .ut4 l Al.ngeteaaa. I lvw.l M I )Davie%. Alt I I.Iilil. .d Ind[duardio TLlert

No. 194 Sotial Ci, i. frlia iFenjiae Edmaio'nti. .4 Stu...%tatdy. K. Sullraraio d.in Laura Ranci

No. 195 Tinyardi a NSutatnaile Dkelvoenirenr 77ir Rio dejatiro, Suliy Edited liv Akira Krciinier. Thcrcz.a Loho. BraizNiticzes., Mohlian Munasinghe, Aind Ronald lParkcr

No. 196 ILarrern Izarope ,le ITraansition: I:ron Resao tl ro ( ria ?1hr. IPsceda u.1 ui a Goniltcreen' oni rite .Ll.acroec,oriat A1.;'nr owtdjLiE3tntr, t,J Cosponired iby tfe Int h tremial .ionetiary Flnd and riat IIiarlld Baink. Edited by Mano I. lilejer.

G;uillermiot A. Calvo. Fabrzio Concieei. and AlazI H. Gclb

No. 19)7 KOTCII lndac.rriai Pol'.ity Igahie3 oftit Pan awr al)d Dirtortmn I;r rire n1-tirue i)e nzD v NI. Leipzigtr and lPeter A. Petrn

No. 19'8 Eporting Mlhla- IExalpe iSir Cuz toda:te. -Saue, Srone IJron l tXlepaa,o ( Coaintrre,. Steven M. Jatec with theasistatitce ol Petcr Gordton

No. 199 Bonmanr (hanenitap .4 dett.r ntenr 1tAramn,n and diie P.uiracal E;'a)mn yiRe;Ijnn. Johli H.Johnson and Sularinan S. WaLity

No. 2111 Soidal Julr astrntazae Co airet" iton in rtie Sahel: htnion. /or npromc Cu Grrenr Prac.tices. Bemard AbeillŽ aindJcan-Matic Lantrani

No. 2111 U 'rbanizartion. .4griacuiaral Develrpimenl. and andS .4Aloaro,t. l)Dpasis uhhadra and Ant5n-io Salazar 1. Brand-o

No. 2112 Ko reanr Indistrial Poliy: Lsyaaies oltite PI'a andl Iaret-iaon., t;.r ruie Futre I )amnv M. Leipzigcr and Petcr A. l'ctn

No. 2413 Pnrver' Reductioll inl f jsr .41ia: 'lue S.hl-rn Rra'oharaou Fndfa Jo hansci

No. 2114 Manlagirni rite Civil Senic e: 7hae Ls,o,s of/ Rrfrn ani Indsrnal Co.airnues. Barbara Nunberg

No. 2115 Desicnting a System of lA hor .arker Saraisric' andl Ifnt nataon. Robert S. Goldfarb and Arvil V. Adanis

No. 2116 Informnation Teclanolocy in ItIorld Bank LUndinrq: Intreasitt the Developiment Imnpat. Nagy Han na anld Sandor Boyson

No. 2117 Proceedinqs of a Conference o Cunrrency Slabhrtiruron and CGrrency Boards. Nissan Liviatan

No. 213I Developing Effective EmploiymentSr Smices. Iavid Fretwell and Susan Goldberg

No. 2119 lEioli,itj L[eal Frametiorkskt;r Pri-atre Sector Developmenr in Central zaJd iiaJren Eiurope. Chcnrl W. Grat and

Associates

No. 21(3 .M-tanec Bioedanoilogfy and Developing Coutnires. Rlynioid A. Zilinskas and Carl Gustaf Lundin

No. 211 Revuratiziriiu Agrnicltural Researcih in rite Salhel: A Pnrposed Frameettvirkafr Actimit . Jan Wcijcnberg. Josu;m D)ions1.

Michacl Fuchs-Garsh. Adolphe KMtn. andJacqucs Leforn

No. 212 Institutional Optionsfor tie Provision of infiastnictitre. Christine Kcssides

No. 213 liae Contribiutions fbInfrastnmacure to E-onotmir Development: A Review e f: Experience and Political Implirations.

Christine Kcssidcs

The World BankHemdquarten European Ofice Tokyo OfficeI 111 II Strert. N W hf. avenue di'1&n.a Kkuimu ilBtding ZWashinblon. D.C- 211433, U S A 7511 b Karim.% Uin 1-I MMrmnutjmi 3 -thonme

(-liv,idajku. 1o,kN) n iii I I pIrtlelione: (2012) 477-1234 Tiephone (1) 4(th') sit (i

F-armile: (21)2) 477-r39'I I-acunlmr: (I) 4(3 (e9 3n (.6 I elephrie (31 3214-S1 NilTelevIx: WU h64145wt miDl PIIANK elAx n4I1R51 latimildl (3) 3214-36.i7

Ii A 2414423 vA't IlI.IIIK FIdrc 2,X3.4Cable Addremi: INlIIAI-IRAI

WA%IIINI. itNIX

7I

2

-I

o

ISBN 0-8213-2650a