DIRECCIÓN DE PLANEACIÓN Y NORMATIVIDAD DE

LA POLÍTICA DE EVALUACIÓN

The Importance of the Institutionalization

of M&E: The Mexican Case

Gonzalo Hernández Licona

2013

Institutional milestones Mexico 1997-2009

1997 Progresa Evaluation 2000 Congress’ Decree: annual external evaluations to

all federal programs 2001 Evaluation Units within ministries 2001 National Audit Office. Congress 2002 First official poverty estimates. Ministry of Social

Development 2004 Law of Transparency and Public Access to Information 2005-6 Social Development Law

CONEVAL. National Council for the Evaluation of Social Policy

2006 Budget Law: Performance Evaluation System 2007 Evaluation Guidelines

CONEVAL, Ministry of Finance, National Audit Office 2009 CONEVAL. Multidimensional Poverty

Methodology 2009 Government Accountability Law

USING THE RESULTS FROM M&E IN SOCIAL POLICY

Changes in social policy due to the M&E System

The government launched in January 2013 an aggressive strategy to reduce hunger and extreme (multidimensional) poverty.

In September 2013 the President sent a bill to Congress to reshape the way social security has been working for almost 50 years.

The fiscal reform of 2013 changed the way resources will be allocated between States and Municipalities.

Since 2009, the Ministry of Finance uses information from the evaluation process to elaborate the annual budget.

There is a constant public and in-house debate and analysis (Presidency, ministries, programs, Congress, media) about the public evaluations of social programs and policies.

CONEVAL published that poverty and food deprivation increased between 2008 and 2012, and the Government acknowledged it.

Multidimensional poverty estimations showed that the larger deprivation is the access to social security (61%)

The reform has the goal to reduce extreme poverty according to the new multidimensional poverty methodology.

There are now annual information for almost 270 social programs. For some of them there are impact, process, design evaluations.

All evaluations are discussed with policy makers and all of them are public.

Changes in social policy due to the M&E System

The budget for the Cement Floor program was increased nearly 400% between 2007 and 2012.

The program Employment for the Youth was cancelled in 2009.

The food supplement of the Progresa-Oportunidades program was modified in order to have a better iron formulae in 2003.

The rural program Procampo is now less regressive, due to a design change in 2009. program were changes due to the evaluation.

Due to the design evaluation of the National Crusade Against Hunger, the strategy improved definitions, diagnosis and coordination tools

The impact evaluation demonstrated reductions of gastrointestinal diseases for children do to the program

The design evaluation of the program showed various design and operational flaws at a considerable cost.

The impact evaluaton showed the iron had not been absorbed by children.

The distributional analysis of various programs illustrated that the program was very regressive.

In July 2013 CONEVAL launched and evaluation to the CNCH, which was discussed with the Ministry of Social Development and also was public. The goal: On going improvement of the strategy.

/1 Information from the Diagnostics of the Indicators Matrixes for Results 2008 and 2010. /2 Information from the Follow-up of Aspects Susceptibles of Improvement System. The estimation considers the aspects susceptibles of improvement attenden by a 100% to march 2012./3 The estimations includes two programas that finished their obligations but are not in operation since 2011.

Various social programs and their budget have changed due to the “Evaluation-Improvement Mechanism”:

Type of Improvement2010/1 2011-2012/2

Programs Relative Participation Programs/3 Relative

Participation

Improvement in programs´ activities or processes 19 16.8% 35 47%

Improvement in the services offered by programs 22 19.5% 10 14%

Reorientation of programs 71 62.8% 26 35%

Programs were merged - - 3 4%

The program was cancelled 1 0.9% 0 0%

TOTAL 113 100.0% 74 100%

CHANGING THE RULES OF THE GAME

Challenges for constructing an Evaluation system:

Institutional: It’s almost impossible to have a public Evaluation system without a proper institutional arrangement: evaluation mandate, evaluation unit, feedback procedures for policy improvements, norms about transparency.

Technical: Suitable methodologies, trained evaluators, administrative records, information.

Constructing an M&E system is a political and institutional task with technical elements….not the other way round….

THE M&E SYSTEM

Identify the main social problems to prioritize strategies and resources.

Public Policy Actions Which programs can

be related to the social priorities?

Improvement of Programs and Policies

• Measuring Poverty (various dimensions)

• Evaluation of Programs and Policies

M&E: Budget decisions based on Results.

Evaluation guidelines for all institutions, together with the Ministry of Finance: The Demand for evidence was clearer now

Ministries’ Strategic objectives

Logical Framework:All Programs

Results

Recommendations’ follow-up

Consistency & ResultsEvaluations

Process Evaluations

Policy Evaluations

Impact Evaluations

Annual Evaluation Plan

Planning Evaluation

NationalDevelopment Plan

Annual Performance Report

Products and facts There are poverty figures at a national, state and municipality level Almost 600 programs have Log Frameworks. All social programs plus

others… We hired ECLAC to help us with the capacity building for the Log

models. We have offered an evaluation course every year for policy makers

and every 2 for researchers. 25% of all indicators are oriented to measure Results 150 programs are evaluated every year (now every two). There are

around 179 social programs. This is 90% of the total budget. 270 one-page summaries every year. Between 3-4 impact evaluations are done every year. CONEVAL budget is around $23 millions on even years (household

survey); $16 millions on odd years. But there is more budget for evaluation within ministries.

We can find on the internet: Poverty estimates All the evaluations The program’s point of view about its evaluation Each program’s Work Plan

ANNEX

Why all these changes in a country where citizens usually mistrust government information and where political parties fight for resources?

Since 1997 there is a balance of power between Congress and the President.

Congress demanded the creation of an independent institution, CONEVAL, for the measurement of poverty and for evaluation.

Information on poverty and evaluation is public and transparent. The Finance Ministry has been an important ally in the process. CONEVAL and the Ministry of Finance produced Guidelines for the

Evaluation of Programs. The center of the Guidelines: Results The Guidelines also created an “Evaluation-Improvement Mechanism”

Most ministries have cooperated in the design of the M&E system. Fine balance between: Accountability and Policy Improvement. The (autonomous) Statistical Office has invested a lot on data

collection. Good academic skills and suitable evaluation methodologies have

been increasingly disposable for the past 15 years.

Institutional milestones Mexico 1997-2007

1997 Progresa Evaluation 1997 Balance of power between Congress and the Executive 2000 Congress’ Decree: annual external evaluations to

all federal programs 2001 Evaluation Units within ministries 2001 National Audit Office 2002 First official poverty estimates2004 Law of Transparency and Public Access to Information 2004-5 Social Development Law

CONEVAL. National Council for the Evaluation of Social Policy

2006 Budget Law: Performance Evaluation System 2007 Evaluation Guidelines

CONEVAL, Ministry of Finance, National Audit Office

CONEVAL Mandate:

Measurement of Poverty at the National, State and Municipality level

Evaluation of social programs and policies

Governance CONEVAL is part of the Executive, but The Board has 8 seats. The majority of Board

members (6) are academic researchers elected by all the States, representatives from Municipalities, Congress and the Executive (44 votes)

Technical and managerial independence

Evidence must be part of a planning system

Identify and measure social challenges

Analysis What works?

Program design Budget Implementation

Monitoring & Evaluation

Why do we need evidence? Improve social

policy

Make better decisions (management, design, budget…)

Accountability

Measuring poverty by Law

Social Developme

nt Law

Dimensions for poverty

measurement

•Current income per capita

• Educational gap

• Access to health services

• Access to social security

• Quality of living spaces

• Housing access to basic services

• Access to food

• Degree of social cohesion

National States

Municipalities

• Multidimensional Poverty Measurement (National and State Estimations)o Economic wellbeingo Education lago Access to health serviceso Access to social securityo Quality and spaces of the

dwellingo Access to basic services in the

dwellingo Access to foodo Social cohesion

• Social Gap Index• Poverty Labor Trend Index

(ITLP)• Poverty Maps• Income Poverty Measurement

• Annual Evaluation Plan• Log Framework: Programs• Consistency & Results

Evaluations• Impact Evaluations• Complementary Evaluations• Performance Evaluations• Programs’ Performance

Summary • Social Development Programs’

Inventory• Thematic Evaluations• Policy Evaluations• Recommendations’ Follow-up• Document for Budgetary

Considerations

Social Programs Evaluation

Poverty Measurement and

Analysis

Main Products

INFORMATION: EXAMPLES

-10.0

-8.0

-6.0

-4.0

-2.0

0.0

2.0

4.0

6.0

-10.0

-8.0

-6.0

-4.0

-2.0

0.0

2.0

4.0

6.0

-10.0

-8.0

-6.0

-4.0

-2.0

0.0

2.0

4.0

6.0

-10.0

-8.0

-6.0

-4.0

-2.0

0.0

2.0

4.0

6.0

Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2008 y 2010

Population whose incomeis below the wellbeing

tresholdAccess to

food

Access to basic servicesin the dwelling

Quality and spaces of the dwelling

Access to social security

Access to health services

Educational gap

Millions of persons

ExtremePovertyPovert

y

200844.5 %

48.8 millions

201046.2 %

52.0 millions

200810.6 %

11.7 millions

201010.4%

11.7 millions

6

4

2

0

-2

-4

-6

-8

-10-9.0

-2.9-2.5

-2.3

-0.8

4.13.5 3.2

0.0

Social Deprivations

4.8

Population whose income is below the minimum wellbeing

treshold

Evolution of multidimesional poverty, Mexico 2008-2010

Percentage of population living in poverty by municipality. Mexico, 2010

Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2010 and the Census 2010

Rangos

[ 0 - 30 ] [ 30 - 50 ] [ 50 - 70 ] [ 70 - 100 ]

Total de municipios

97 347 790 1222

Ranks

Total municipalities

Programa de Empleo Temporal (PET)

NA Moderate Adecuate Adecuate Adecuate 379.59% 92.5%The program is

VERY PROGRESSIVE

99.5%

Programa IMSS-Oportunidades

NA Adecuate Adecuate Adecuate Moderate SD 90.0%The program is

VERY PROGRESSIVE

100.0%

Seguro Popular (SP) Adecuate NA Adecuate Adecuate Outstanding 88.54% 100.0%The program is

VERY PROGRESSIVE

100.0%

Programa Comunidades Saludables

NA Adecuate Moderate ModerateOpportunity for Improvement

SD 80.0%Without

Information100.0%

Programa Caravanas de la Salud (PCS)

NA Moderate Adecuate Adecuate Moderate SD 100.0%Without

Information100.0%

Reducción de Enfermedades Prevenibles por Vacunación

NAOpportunity for Improvement

Opportunity for Improvement

OutstandingOpportunity for Improvement

SD NAWithout

Information100.0%

PROCAMPO para Vivir Mejor NA Adecuate Adecuate Adecuate Adecuate 99.43% 75.0%The program is

VERY REGRESSIVE

100.0%

Fondo de Apoyo para la Micro, Pequeña y Mediana Empresa (Fondo PYME)

Adecuate NA Outstanding Outstanding Outstanding 150.30% 69.0%Without

Information100.0%

Progressivity Level

Spent Budget/ Modified Budget

Program

Valuation

Improvements on the Delivery of Goods and

Services

Improvements on Indicators

and Goals Analysis

Coverage Coverage Efficiency

% of Achievement on

Following the Recommendations from External

Evaluations

Impact of the Program

Improvements on the

Achievement of its Objectives

RESULTS FROM THE SPECIFIC PERFORMANCE EVALUATION (EED) 2010-2011(External evaluation coordinated by CONEVAL and elaborated with information from the Performance Evaluation System of the

Ministry of Finance and Public Credit) Distributional Impact

2010 Budget Compliance

Results Related to the Program´s Objectives

Evaluation:Programs’ Performance Summary

CHALLENGES

Better coordination between evaluation institutions and budget allocation decisions based on results: Planning vs Budget; CONEVAL, Ministry of Finance and Ministry of Audit

More capacities on evaluation and measurement tecniques

Improve indicators of diverse programs Indicators of quality of educational and health services

List of beneficiaries Evaluation of legislation and norms’ changes Evaluation of Sectors/Ministries. Strategic objectives. Rigorous evaluations and transparency in states

and municipalities. Take more into account the information from

evaluations in budgetary, operative and strategic decisions.

Challenges for Mexico

SOME LESSONS

Lessons Countries should find their own institutional arrangement: Chile,

Colombia, South Africa, USA, Canada, Mexico, China, etc. The balance of power between Congress and the Executive is

important Public pressure helps

Credibility is at the center of the institutional arrangement If the country engaged in specific evaluations previously, use

the experience to set up a system in the future Champions are always a key factor Internacional help is important…but try to build your own

path Impact evaluations are not always the first step, more basic

evidence is sometimes more important South-South exchanges of knowledge and innovation Keep the fine balance between Transparency and Policy

Improvement. On of the main users: Ministry of Finance