11

Monitoring and Evaluation (M&E) Systems to Improve Government Performance

Presentation to the PEMPAL Plenary Conference

Istanbul, 27 February 2008

Keith MackayIndependent Evaluation Group

The World Bank(kmckay@worldbank.org)

22

Why Countries Want an M&E Why Countries Want an M&E System System

To support budget decision-making, To support budget decision-making,

i.e. performance-based budgeting (PBB)i.e. performance-based budgeting (PBB)

3 main types: direct, indirect, presentational3 main types: direct, indirect, presentational

To support national, sectoral and sub-national To support national, sectoral and sub-national planningplanning

To design policies and programsTo design policies and programs

To assist ministries, entities and sub-national To assist ministries, entities and sub-national governments in their managementgovernments in their management

To strengthen accountability relationshipsTo strengthen accountability relationships

33

Related Uses of M&E InformationRelated Uses of M&E Information

To clarify government goalsTo clarify government goals To set performance targets; customer service To set performance targets; customer service

standardsstandards To contract public services to private sectorTo contract public services to private sector Performance contracts; personnel appraisalPerformance contracts; personnel appraisal Anti-corruption; measuring “leakage” of Anti-corruption; measuring “leakage” of

government fundsgovernment funds Civil society oversight of government Civil society oversight of government

performance “voice”performance “voice” EtcEtc

44

Government M&E Systems: Government M&E Systems: Possible Roles of a MoFPossible Roles of a MoF

1.1. Manager of the M&E systemManager of the M&E system

2.2. StakeholderStakeholder

3.3. Initiate evaluations ― choice of programs, Initiate evaluations ― choice of programs, terms of referenceterms of reference

4.4. Manage / commission evaluationsManage / commission evaluations

5.5. User of M&E info ― for direct, indirect, User of M&E info ― for direct, indirect, presentational PBBpresentational PBB

6.6. Funder of evaluationsFunder of evaluations

7.7. Passive non-userPassive non-user

8.8. RoadblockRoadblock

55

PBB ― What Benefits For a MoF?PBB ― What Benefits For a MoF?

1.1. Reorient budget process from emphasis on $ Reorient budget process from emphasis on $ to also emphasize results ― outputs, to also emphasize results ― outputs, outcomes, impacts of spendingoutcomes, impacts of spending

2.2. M&E information provides a new source of M&E information provides a new source of options for budget reallocationsoptions for budget reallocations

3.3. (M&E) “information is power” ― conversely, a (M&E) “information is power” ― conversely, a danger of information assymetry if sector danger of information assymetry if sector ministries only have this informationministries only have this information

66

PBB ― Challenges For a MoF to PBB ― Challenges For a MoF to ConsiderConsider

1.1. M&E can be costly ― who should pay?M&E can be costly ― who should pay?

2.2. Requires special skills / expertise to manage Requires special skills / expertise to manage an M&E system and to conduct or contract out an M&E system and to conduct or contract out evaluationsevaluations

3.3. Budget timing versus duration of evaluationsBudget timing versus duration of evaluations

4.4. Need for budget analysts to become Need for budget analysts to become discriminating consumers of M&E infodiscriminating consumers of M&E info

5.5. Can be difficult to reorient budget analysts Can be difficult to reorient budget analysts from spending to performancefrom spending to performance

77

What is “M&E”?What is “M&E”?

Performance indicators inputs, processes, Performance indicators inputs, processes, outputs, outcomes, impactsoutputs, outcomes, impacts

Clarify program logic / results chainsClarify program logic / results chains

Rapid evaluationsRapid evaluations

Rigorous impact evaluationsRigorous impact evaluations

Cost-benefit analysisCost-benefit analysis

Etc many tools, methods, approachesEtc many tools, methods, approaches

Main dichotomy is between monitoring (i.e. Main dichotomy is between monitoring (i.e. performance indicators) and evaluationperformance indicators) and evaluation

Each M&E tool has strengths, costs, limitationsEach M&E tool has strengths, costs, limitations

88

Country Case Study:Country Case Study: Chile Chile

M&E system mainly designed and used by MoFM&E system mainly designed and used by MoF Ex ante cost-benefit analysis for investment Ex ante cost-benefit analysis for investment

programs (started by Planning Ministry in 1975)programs (started by Planning Ministry in 1975) 1,550 performance indicators (1994) for all 1,550 performance indicators (1994) for all

government programsgovernment programs ““Program evaluations” (1996): desk reviews of Program evaluations” (1996): desk reviews of

logframe: 10-12 each year, take 4-6 months, logframe: 10-12 each year, take 4-6 months, average cost = $11,000average cost = $11,000

Rigorous impact evaluations (2001): 4 each year, Rigorous impact evaluations (2001): 4 each year, take <18 months, average cost = $88,000take <18 months, average cost = $88,000

Comprehensive Spending Reviews (2002): desk Comprehensive Spending Reviews (2002): desk reviews of all programs in a functional area: 1-2 reviews of all programs in a functional area: 1-2 each year, cost = $48,000each year, cost = $48,000

99

Chile’s M&E System Chile’s M&E System Strengths (i) Strengths (i)

Performance information used to identify need Performance information used to identify need for evaluationsfor evaluations

Evaluations conducted externally, in fully Evaluations conducted externally, in fully transparent process, and are highly credibletransparent process, and are highly credible

All M&E findings reported publicly and sent to All M&E findings reported publicly and sent to CongressCongress

M&E system closely linked to the budget M&E system closely linked to the budget information needs of MoFinformation needs of MoF

Performance information used to set targets for Performance information used to set targets for ministries – these are largely metministries – these are largely met

MoF closely monitors extent of utilization of MoF closely monitors extent of utilization of evaluation findingsevaluation findings

1010

Chile’s M&E System Chile’s M&E System Strengths (ii) Strengths (ii)

High utilization of M&E findings by MoF in the High utilization of M&E findings by MoF in the budget process and to impose management budget process and to impose management improvements on ministries, agenciesimprovements on ministries, agencies

Utilization of government evaluations 2000 to 2006Utilization of government evaluations 2000 to 2006

Minor Minor adjustment adjustment of program, of program, for example for example improved improved information information systemsystem

Major change Major change in mgment in mgment processes e.g. processes e.g. new targeting new targeting criteria, new criteria, new management management information information systemssystems

Substantial Substantial redesign of redesign of program or program or of organiz-of organiz-ational ational structurestructure

Institutional Institutional relocation relocation of programof program

Program Program terminationtermination

TOTALTOTAL

24%24% 38%38% 25%25% 5%5% 8%8% 100%100%

1111

Chile’s M&E System Chile’s M&E System Weaknesses Weaknesses

Unevenness in quality of evaluations – due to Unevenness in quality of evaluations – due to cost and time constraintscost and time constraints

Chile probably not spending enough on Chile probably not spending enough on evaluations: only $750,000 each year evaluations: only $750,000 each year compared with total government budget of $20 compared with total government budget of $20 BillionBillion

Low utilization – low ‘ownership’ – of MoF’s Low utilization – low ‘ownership’ – of MoF’s evaluations by sector ministriesevaluations by sector ministries

1212

Country Case Study:Country Case Study: AustraliaAustralia

Context: fiscal crisis Context: fiscal crisis therefore many public sector and budget reforms; therefore many public sector and budget reforms; autonomy given to line departments, agenciesautonomy given to line departments, agencies

M&E system managed by Dept of Finance (DoF)M&E system managed by Dept of Finance (DoF)

Evaluations mandatory – every 3-5 years for Evaluations mandatory – every 3-5 years for every programevery program

Sector departments had to prepare rolling, 3-Sector departments had to prepare rolling, 3-year plans for evaluationsyear plans for evaluations

Broad range of evaluation types ― rapid Broad range of evaluation types ― rapid evaluations; formal reviews; cost-benefit evaluations; formal reviews; cost-benefit analysis; impact evaluations; performance analysis; impact evaluations; performance audits; etcaudits; etc

1313

Australia’s M&E System (1987-1997) Australia’s M&E System (1987-1997) Strengths Strengths

Involvement of DoF budget / policy analysts in Involvement of DoF budget / policy analysts in evaluation planning and major evaluationsevaluation planning and major evaluations

By mid-1990s, ∑160 evaluations underway at By mid-1990s, ∑160 evaluations underway at any point in timeany point in time

Evaluation findings heavily used in budget Evaluation findings heavily used in budget analysis, policy advice, and also by the Cabinet analysis, policy advice, and also by the Cabinet in its budget decision-makingin its budget decision-making

High utilization of evaluation findings by sector High utilization of evaluation findings by sector departments and agenciesdepartments and agencies

Evaluation was a collaborative endeavor Evaluation was a collaborative endeavor between Finance Department, other central between Finance Department, other central departments, and sector departmentsdepartments, and sector departments

1414

Australia’s M&E System (1987-1997) Australia’s M&E System (1987-1997) Weaknesses Weaknesses

Uneven quality of evaluationsUneven quality of evaluations

Insufficient attention to regular performance Insufficient attention to regular performance informationinformation

Insufficient central support for advanced Insufficient central support for advanced evaluation training evaluation training

An administrative burden on departments An administrative burden on departments was claimedwas claimed

1515

Australia’s Second Generation M&E Australia’s Second Generation M&E System (1997-2006)System (1997-2006)

Genesis was new conservative government: Genesis was new conservative government:

significant reduction in size of civil servicesignificant reduction in size of civil service

policy-advising system largely dismantledpolicy-advising system largely dismantled

““bureaucratic” rules and central oversight bureaucratic” rules and central oversight reduced considerablyreduced considerably

evaluation strategy dismantled: evaluation was evaluation strategy dismantled: evaluation was “deregulated”“deregulated”

““Performance framework” – emphasis on Performance framework” – emphasis on performance monitoringperformance monitoring

1616

Australia’s M&E System (1997-2006)Australia’s M&E System (1997-2006)

New M&E system based on mix of principles, New M&E system based on mix of principles, expectations, some requirements expectations, some requirements

Emphasized performance reporting to Emphasized performance reporting to Parliament, both ex ante and ex postParliament, both ex ante and ex post

Evaluation was encouraged ― but not requiredEvaluation was encouraged ― but not required

DoF was downsized considerably. Its role in DoF was downsized considerably. Its role in providing advice during budget process and for providing advice during budget process and for maintaining budget estimates was reduced maintaining budget estimates was reduced considerablyconsiderably

1717

Australia’s M&E System (1997-Australia’s M&E System (1997-2006): How Successful? 2006): How Successful?

National Audit Office: departments’ reports to National Audit Office: departments’ reports to Parliament inadequate:Parliament inadequate: poor quality data ―weak standards, systemspoor quality data ―weak standards, systems little use made of targets or benchmarkinglittle use made of targets or benchmarking lot of data on government outputs, but little on lot of data on government outputs, but little on outcomesoutcomes lack of real analysis of performance informationlack of real analysis of performance information Parliamentary committees very unhappy with this Parliamentary committees very unhappy with this inadequate informationinadequate information

OECD (2002): The changes to the M&E system OECD (2002): The changes to the M&E system “deprived the Finance Ministry of the information “deprived the Finance Ministry of the information necessary for it to adequately advise the Minister”necessary for it to adequately advise the Minister”

1818

Lessons for ECA Countries (i)Lessons for ECA Countries (i)

Lessons from these 2 countries are consistent Lessons from these 2 countries are consistent with other countries’ experiencewith other countries’ experience

1.1. Key role of powerful champion of M&E Key role of powerful champion of M&E ideally centrally-driven, by capable ideally centrally-driven, by capable

ministry such as a MoFministry such as a MoF

2.2. Demand and supply sides both need Demand and supply sides both need careful careful attentionattention

3.3. On demand side, incentives are key to On demand side, incentives are key to achieve high utilizationachieve high utilization

in Australia it took several years to achieve in Australia it took several years to achieve cultural change: required strong leadership; cultural change: required strong leadership;

focused recruitment / promotion; on-the-job focused recruitment / promotion; on-the-job training; staff turnovertraining; staff turnover

1919

Lessons for ECA Countries (ii)Lessons for ECA Countries (ii)

4.4. Monitoring information (indicators) are Monitoring information (indicators) are useful – but important to have evaluations to useful – but important to have evaluations to understand understand reasonsreasons why performance is why performance is good or badgood or bad

5.5. Monitoring information and evaluation Monitoring information and evaluation findings need careful analysis to be usefulfindings need careful analysis to be useful

requires appropriate skills and staffingrequires appropriate skills and staffing

6.6. On supply side, (i) caveat multiple, On supply side, (i) caveat multiple, competing, uncoordinated systems; (ii) competing, uncoordinated systems; (ii) ensure data reliability, credibilityensure data reliability, credibility

2020

Lessons for ECA Countries (iii)Lessons for ECA Countries (iii)

7.7. Role of structural arrangements to ensure Role of structural arrangements to ensure M&E objectivity and quality:M&E objectivity and quality:

Who is responsible for planning, conduct and Who is responsible for planning, conduct and use of evaluations? use of evaluations?

Self-evaluations or independent evaluations?Self-evaluations or independent evaluations?

Who funds the system? ― (a) earmarked Who funds the system? ― (a) earmarked budget funds?; (b) costs “absorbed” by budget funds?; (b) costs “absorbed” by ministries?ministries?

Linking M&E information and financial Linking M&E information and financial management information systems (FMIS); management information systems (FMIS);

program budgeting program budgeting

2121

Lessons for ECA Countries (iii)Lessons for ECA Countries (iii)

8.8. Long-haul effort, requiring patienceLong-haul effort, requiring patience

9.9. Limitations of relying on laws, decrees, Limitations of relying on laws, decrees, regulationsregulations

10.10. There are no strong pre-conditions, but There are no strong pre-conditions, but success factors include:success factors include:

program budgetingprogram budgeting

reliable FMISreliable FMIS

reliable ministry data systemsreliable ministry data systems

powerful and capable MoFpowerful and capable MoF

government Ministers and Cabinet who government Ministers and Cabinet who support evidence-based policy-making support evidence-based policy-making

2222

Useful Resource Materials Useful Resource Materials

Keith Mackay, Keith Mackay, How to Build M&E Systems for Better How to Build M&E Systems for Better GovernmentGovernment, World Bank, 2007., World Bank, 2007. http://www.worldbank.org/ieg/ecd/better_government.htmlhttp://www.worldbank.org/ieg/ecd/better_government.html

Marcela Guzman, Marcela Guzman, Systems of Management Control and Systems of Management Control and Results-Based Budgeting: The Chilean ExperienceResults-Based Budgeting: The Chilean Experience, Ministry , Ministry of Finance, Chile, 2003.of Finance, Chile, 2003.http://www.dipres.cl/control_gestion/publicaciones/Sistems_of_management_control_and_results-based_budgeting.htmlhttp://www.dipres.cl/control_gestion/publicaciones/Sistems_of_management_control_and_results-based_budgeting.html

Keith Mackay, Keith Mackay, Two Generations of Performance Evaluation Two Generations of Performance Evaluation and Management System in Australiaand Management System in Australia, World Bank, 2004., World Bank, 2004. http://lnweb18.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/http://lnweb18.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/14163969A1A709BD85256E5100013AA8/$file/ecd_wp_11.pdf14163969A1A709BD85256E5100013AA8/$file/ecd_wp_11.pdf

Ernesto May et al. (eds.), Ernesto May et al. (eds.), Towards the Institutionalization of Towards the Institutionalization of Monitoring and Evaluation Systems in Latin America and Monitoring and Evaluation Systems in Latin America and the Caribbeanthe Caribbean, World Bank/IADB, 2006., World Bank/IADB, 2006. http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/LACEXT/0,,contentMDK:20893139~pagePK:146736~piPK:146http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/LACEXT/0,,contentMDK:20893139~pagePK:146736~piPK:146830~theSitePK:258554,00.html830~theSitePK:258554,00.html

2323

Useful Resource Materials Useful Resource Materials

Teresa Curristine, Teresa Curristine, Performance Information in the Budget Performance Information in the Budget Process: Results of the OECD 2005 QuestionnaireProcess: Results of the OECD 2005 Questionnaire, in , in OECD Journal on Budgeting, 2005 5(2),pp.87-131. OECD Journal on Budgeting, 2005 5(2),pp.87-131. http://www.oecd.org/document/38/0,3343,en_2649_34119_39921702_1_1_1_1,00.htmlhttp://www.oecd.org/document/38/0,3343,en_2649_34119_39921702_1_1_1_1,00.html

OECD, OECD, Performance Budgeting in OECD CountriesPerformance Budgeting in OECD Countries, OECD, , OECD, 2007. 2007. http://www.oecd.org/document/38/0,3343,en_2649_37405_39921702_1_1_1_37405,00.html#B2http://www.oecd.org/document/38/0,3343,en_2649_37405_39921702_1_1_1_37405,00.html#B2

Independent Evaluation Group, Independent Evaluation Group, Monitoring & Evaluation Monitoring & Evaluation Tools, Methods and ApproachesTools, Methods and Approaches, World Bank, 2004. , World Bank, 2004. http://http://www.worldbank.org/ieg/ecd/me_tools_and_approaches.htmlwww.worldbank.org/ieg/ecd/me_tools_and_approaches.html

Independent Evaluation Group, Independent Evaluation Group, Influential Evaluations: Influential Evaluations: Evaluations that Improved Performance and Impacts of Evaluations that Improved Performance and Impacts of Development ProgramsDevelopment Programs, World Bank, 2004., World Bank, 2004.http://www.worldbank.org/ieg/ecd/influential_evaluations.htmlhttp://www.worldbank.org/ieg/ecd/influential_evaluations.html

World Bank website on Building Government M&E World Bank website on Building Government M&E Systems:Systems: www.worldbank.org/ieg/ecdwww.worldbank.org/ieg/ecd//