1

Building Statistical Capacity to monitor the Progress of Societies

Techniques for Evaluating Public Policies in Developing Countries

Luiz Awazu Pereira da Silva

Ministry of Budget and Planning (Brazil)OECD World Forum on Statistics, Knowledge and Policy on

Measuring and Fostering the Progress of Societies

Istanbul, Turkey from 27 to 30 June 2007Based on:

“The Impact of of Economic Policies on Poverty and Distribution, Volume 1” by François Bourguignon, Luiz A. Pereira da Silva, eds. The World Bank, Oxford University Press (2003), and Volume 2 with Maurizio Bussolo (ed.) forthcoming (2007)

2

Measuring Progress of Societies changed from 50s (NA, aggregate data) to 90-00s specific economic policies and

structural reforms in DCs (HHS, micro data)

1. Even macro-economic policies, monitoring post-stabilization requires specific surveys, not just NA aggregate dataFiscal monetary policy stance, exchange rate regime, public debt management strategy, etc. role of expectations (from market surveys) and of high frequency specific data (daily)

2. Evaluation of efficiency of public expenditures design of social policies requires targeting thru micro dataTax policy reform, composition of public expenditure, design of social programs (CCTs) civil service reform, pension reform, decentralization

3. Ex-ante analysis of structural reforms tooTrade liberalization, liberalization of specific markets, financial sector reforms, improving the investment climate, land reform, privatization etc.

3

30,4 33,6 33,2

25,3 24,5 23,5 22,0

18,0 17,9 16,8 17,1

9,7 11,3 10,7

9,0 9,4 9,5

33,6

10,1

8,97,15,85,07,6

0,0

10,0

20,0

30,0

40,0

50,0

60,0

70,0

80,0

90,0

100,0

2002 2003 2004 2005

Outras Discricionárias

Despesas Saúde eEducação

Outras Obrigatórias

Transferências a Estadose Municípios

Despesas de Pessoal

DespesasPrevidenciárias

3.5

3.75

4

4.25

4.5

28-O

ct-02

12-N

ov-0

2

27-N

ov-0

2

12-D

ec-0

2

27-D

ec-0

2

11-J

an-0

3

26-J

an-0

3

10-F

eb-0

3

25-F

eb-0

3

12-M

ar-0

3

27-M

ar-0

3

11-A

pr-0

3

26-A

pr-0

3

11-M

ay-0

3

26-M

ay-0

3

0

2

4

6

8

10

12

14

16

Expected Primary Fiscal Surplus (Public Sector) for 2003 % of GDP (left scale)

Expected Inflation (IPCA) for 2003, % p.a. (right scale)

-6.00

-5.00

-4.00

-3.00

-2.00

-1.00

0.00

1.00

2.00

3.00

1999

*

2000

*

2001

*

2002

*

2003

*

2004

*

2005

*

2006

*

Sub-National Public Sector Federal Non-Financial Public EnterprisesCentral Government Social SecurityConsolidated Public Sector Target for Primary Fiscal Surplus

1998-99 Currency

Crisis&

Adjustment

2002-2003 Credibility Crisis &

Adjustment

2005 Preparing for electoral year

Measuring Progress in Brazil: New Data and Surveys (Monthly, Daily to monitor macro policies and efficiency of public spending)

a) Credibility of Macro-Policies need to be monitored by high frequency macro data from specific surveys (daily, weekly)

b) Where is really going public spending (40% of GDP of about $1trillion), especially health, education, social security, requires HHS and micro data

4

p

inflation

w wages

L employt.

Y growth

Gpublic

spending.

Macro data, aggregated, NA and specific surveys

Measuring progress with macro economic policies

• Aggregate and first principle analysis used: assuming homogeneity of individuals and households, adds up into aggregate or “average” impact (e.g., higher growth, higher employment, lower inflation, higher social spending are all “good” on average for progress of societies)

• Policy objectives essentially macro concepts (e.g., IMF, World Bank of the 1960-70s, “old” Governments, etc.)

• Macro data bases (NA) available as the natural analytical environment for aggregate, growth/social analysis

5

• Scope and objective, challenges increase: evaluate the economic feasibility of public programs and policies and their overall ‘development” impact.

• Aggregate and first principle analysis insufficient: heterogeneity of individuals and households, microeconomic behavior do not add up into aggregate nor “average”, specificities of economic structures and local political economy, transmission of shocks and policies

• Policy objectives and social demand increasingly focusing on distributional effects and poverty reduction, essentially micro concepts (e.g., Post-WC IFIs, new types of Governments, etc.)

• Micro data bases (household surveys HHS) increasingly available as the natural analytical environment for distributional and poverty analysis

Measuring progress with social policies

6

);(/);(

)0()1(:

pCPAELwRELwy

PyPyy

iiiiiiiii

iiiii

yi : real income

wi : wage rate

Li : labor supply

Ei : self-employment, non-wage income

Ri : net transfer income

Ai : socio-economic characteristics

Ci : consumption characteristics

household-specific P price indexp : general price index

Measuring progress in social policiesDefine impact for individual i as the difference in income yi with and

without the program, denominated yi :

7

Household Survey (HHS), i individual households

Compare the distribution of y|P=1 with the distribution of y|P=0.Calculate changes in inequality or poverty across the two distributionsDifferent tools/methods differ in how they construct the counterfactual distribution and the data that are neededRank results according to some agreed upon rule and/or objective

p

prices

w wage

L employt.

R transfers

A households character.

)0()1(:

);(/);(

iiiii

iiiiiiiii

PyPyy

pCPAELwRELwy

Program or policy will shock one or more components that explain theindividual income yi

Evaluation of Program and/or Economic Policy

8

Several “Microeconomic techniques”

1. Average Incidence Analysis a) Tax Incidence Analysis (Sahn & Younger)

b) Public Expenditure Incidence Analysis (Demery)

2. Marginal Incidence Analysisa) Behavioral response to changes (Van de Walle)

b) Poverty mapping (Lanjouw)

3. Impact Evaluation (randomization, matching, double-dif)a) Ex-post (Ravallion)

b) Ex-ante (Bourguignon & Ferreira)

4. Data and Measurement (not covered here)

a) Multi-topic Household Surveys (Scott)

b) Qualitative surveys (Rao & Woolcock)

c) Performance in Service Delivery (Dehn, Reinikka & Svensson)

9

468 - 1707.21707.2 - 2216.12216.1 - 2639.22639.2 - 4040.84040.8 - 13777.3

Madagascar: Per capita health spending per district

Madagascar: Incidence of Poverty per district

0.191 - 0.5660.566 - 0.6960.696 - 0.7660.766 - 0.8330.833 - 0.946

10

New “Macroeconomic techniques”, from robust to more speculative….

1. Standard RHG approaches to macro-micro linkage: a) "Micro-accounting"/RHG approach based on aggregate macro predictions

(PovStat-SimSip-PAMS)

b) The disaggregated SAM-CGE/RHG approach (Adelman-Robinson, Bourguignon and al. in the "Maquette“, Loefgren, Robinson or Agenor and al. with IMMPA.)

2. Top-down "micro-simulation" approaches (micro-macro linkages)a) "Micro-accounting modules" linked to disaggregated macro models (Chen-

Ravallion, McCulloch-Winters)

b) "Micro-simulation modules" linked to disaggregated macro models (Bourguignon-Robilliard-Robinson, Ferreira-Leite-Pereira-Picchetti, Cogneau-Robilliard-van der Mensbrugghe)

3.Other issues for research and applications:

a) Fully integrated models (Townsend, Heckman, Browning-Hansen-Heckman)

b) Accounting for general equilibrium effects of public expenditure programs

c) Dynamic modeling and the proper treatment of growth

11

Household Survey (HHS), i individual households

iii

iiiiiiiii

yyy

pCPAELwRELwy

1,

);(/);(

p

prices

w wage

L employt.

R transfers

A households character.

Macro framework, general/partial equilibrium

Evaluation of macro economic policies.Macro to micro linkages

Instead of « exogenous and independent » shocks like before use « endogenous and  dependent» shocks to 'microsimulate' the effect of policies on all individuals in the

micro data sets, and the poor some consistency constraints will be « binding » (e.g., budget envelope for social programs, real GDP growth, etc.)

LAVs =Linkage Aggregate Variables

12

Top-down "micro-simulation" approach within a macro-micro linkage approach

Macro model

Linkage AggregatedVariables

(prices, wages, employment levels)

Household income micro-simulation model

13

Top-down "micro-simulation" approach within a macro-

micro linkage approach

Two distinct approaches to micro-module:

- "micro-accounting" : no explicit change in behavior (envelope theorem argument), e.g., Chen-Ravallion

- "micro-simulation" : change in behavior, possibly linked to (labor) market imperfections, e.g., Robillard, Bourguignon, Robinson and Ferreira, Leite, Pereira da Silva, Picchetti

LAVs from above

Household income micro-simulation model

14

Example of Brazil Results using this technique: Aggregate Poverty and Inequality Indices (on

aggregate, good results) after financial crisis 1999

Departure: Actual data from the HHS (PNAD)

Simulated by the Macro & Micro

Models (B)

Comparison: Actual data from the HHS (PNAD)

(a) (b) (c) (b)/(c)P0 Poverty Headcount

28.1% 29.8% 29.4% 1.013

P1 Poverty Gap 11.6% 12.6% 12.3% 1.026

p2 6.5% 7.1% 6.9% 1.037

e0 0.662 0.655 0.649 1.010

e1 0.715 0.695 0.695 1.000

e2 1.731 1.630 1.563 1.043

Gini 0.593 0.588 0.588 1.001

Mean of HPCI in R$ / month

257.30 260.46 257.75 1.011 (0.9971)

note 1: ratio computed using real terms

Brazil - Nominal Terms

15

Example: Brazil, 1999 Financial Crisis, Results of Simulationnominal changes in per capita income after floating ER, across centiles of income

Figure 7 - Comparison betweenActual Observed Changes &

Experiment 1 - using Representative Households Groups (RHG)Experiment 2- using Pure Micro Simulation model

Experiment 3 - using Full Macro-Micro Linkage model

-0.12

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0 10 20 30 40 50 60 70 80 90 100

Percentiles

Lo

g d

iffe

ren

ce

Actual Experiment 1 - RHG

Experiment 2 - Pure Micro Simulation Experiment 3 - Full Macro-Micro Linkage

Percent changes between 1999 and 1998 in Nominal Income (in Reais, R$) / Month for each percentile of the distribution in Brazil

16

Top-down "micro-simulation" approach

Macro model

Linkage AggregatedVariables

(prices, wages, employment levels)

Household income micro-simulation model

Final Remarks: introducing feedbacks

Feedback, e.g., micro-transfers, minimum wage

17

Concluding Remarks1. Statistical capacity is needed more than ever and evolves with

changes in policy-making practices and objectives (from aggregate growth to local quality & efficiency of public spending, targeting, evaluation ex-post/ex-ante)

2. Data needed evolved from aggregate-macro (National Acc.) to sectoral and mostly micro (HHS); continuity of time series is paramount to measure policy results

3. Information technology for collecting data, lower costs, allows applying techniques to MICs and LICs; international coordination (OECD, WB, etc.) important for cross-country and panel comparisons with policy-implications

4. Statistical capacity, data collection needs to be built-in project and policy design, ex-ante and not only ex-post

5. Macro policies need continuous micro evaluation of results to measure progress/efficiency (e.g., Bolsa-Familia in Brazil)

18

Measuring Progress in Society?

Output Growth, quantitative indicators

Effectiveness ofPublic Spending,

RedistributionPoverty Reduction

“Kuznets”type-curveAggregate growththen redistribution

“Effective social policies”for growth

and redistributionrequires evaluationmicro-data (HHS)

& adequate policy design

19

END

Caminante no hay camino, se hace camino al andar…

(Antonio Machado)

20

Question? Usage of the techniques to meet DCs policy challenges, if, when,

where and how• Are these techniques for evaluation of public policies used? Some of

these techniques are 'simple', yet not all of them are widely used, why? Costs (training) and institutional implications for Ministries and agencies (Finance vs. Planning, political economy of budget process, etc…). Aid agencies (IDA, DFID, AFD, etc…) promoting evaluation,

• When these techniques are used, are they useful for policy-makers? They capture only the 'between' (RHG) dimension of distributional changes, which empirically proves limitative. They are ill adapted to the distributional aspects of growth, but important in putting broader (poverty) perspective to decisions

• Where are these techniques for evaluation of public policies used? Examples below

21

Usage of these “techniques” in connection with policy-making (with external assistance)

Average Incidence Analysis (Tax Incidence Analysis and Public Expenditure Incidence Analysis)

Most OECD countries. Also in many DCs particularly IDA [Ghana (ISSER), Madagascar (INSTAT & Cornell Univ.), Uganda (EPRC)]

Marginal Incidence AnalysisMany OECD countries and India, using NSS 1994; Indonesia, using

SUSENAS 1981 & 1987; Vietnam using panel from VLSS 1993 & 1998; Argentina,using public spending and census data, Ministry of labor team; Brazil using PNADs

Poverty mapping Many OECD countries Ecuador, Bolivia, Mexico, Panama, Nicaragua, Guatemala, South Africa, Madagascar, Kenya, Uganda, Malawi, Mozambique, Tanzania, Bulgaria, Albania, Thailand, Vietnam, Cambodia, Indonesia, China., Brazil, etc.

Ex-post impact evaluation methods (randomization, PSM, double-dif)Many OECD countries. Also in many DCs, Argentina, Brazil, Kenya

22

"Micro-accounting"/RHG approach based on aggregate macro predictions SimSIP/PAMS

Latin America, Burkina-Faso, Thailand, Indonesia

The disaggregated SAM-CGE/RHG approach

IFPRI (US) country models, IMMPA-Cameroon, Brazil, many countries have GTAP+LAVs+HHS

"Micro-accounting modules" linked to disaggregated macro models China, Colombia, Brazil, Many countries have GTAP+LAVs+HHS

"Micro-simulation modules" linked to disaggregated macro models Indonesia, Brasil

More sophisticated?

Fully integrated models (Thailand, Madagascar)

Accounting for general equilibrium effects of public expenditure programs (???)

Dynamic modeling and the proper treatment of growth (???)