1

Practical Issuesin Applying CIE

Daniele BondonioUniversity of Piemonte Orientale

2

•Estimating average treatment effect for all the treated units is often non-informative….”black box evaluation”

•Relevant policy issues can be incorporated in CIE by estimating different impacts for different types of treated units or different types of interventions

•E.g. [enterprise support] different impacts can be estimated for:

-soft loans, grants, technical assistance-different economic values of the subsidies-different types of assisted firms (small,

large)(textile, chemicals, …services,…..)

How to incorporate policy relevant issues in CIE

3

•E.g. job training: -different ages of participants-different education levels-previous work experience

•Urban revitalization:-different degrees of initial distress

•Tourism promotion programmes:-cultural events-infrastructure improvements-urban renovations

4

When different impacts are estimatedfor different groups of treated units &/or different variations of how the

treatment is providedCIE can offer insights on how a programme works

5

•E.g. for enterprise support, local economic development ……multiple different programs are frequently available and affecting a same outcome variable of interest

(firm sales, productivity, employment, investments…. indicators of quality of life from

survey-data……)

•Caution with single-programme evaluations using outcome variables that can be affected by other programmes not included in the analysis

Single-programme vs multiple- programme evaluations

6

I)Solution: try to include all programmes in your analysis (e.g. enterprise support taking into account multiple programs)

e.g. enterprise support: estimate the impact of different economic intensity of support, different types of incentives

II) Solution: try to focus on intermediate outcomes that are affected solely by the programme

Y=number of tourism visits instead of more distant outcomes like local employment (for tourism promotion programmes)

Y= innovation outcomes instead of sales growth, productivity…(for R&D support to enterprises)

7

Measuring the timing of the intervention

•E.g. enterprise support. Data on program incentives

-dates on when the application was approved-dates on incentive payments (1 installment, 2 installment…..)

Which dates are to be used in measuring the beginning of the treatment?

•Overlooked issue with very strong consequences on impact estimates

-when should we locate T?Time

• • • Approval 1st payment 2nd payment

8

•If the programme support is wrongly placed in a time earlier than the time in which the outcome of interest could be potentially affected, the outcomes of such time would be erroneously considered as exposed to the treatment

• By contrast, if a programme intervention is wrongly placed in a time later than the time in which the outcome of interest could be potentially affected, the outcomes of such period would be erroneously considered as not-exposed to the treatment

9

•Percentage change is often used:

%=

•% adequate for many CIE (e.g. unit of observations: individuals in need of assistance, geographic areas)

•% to be used with caution for enterprise support: the outcomes produced by the program (against the estimated counterfactual) has a social utility which may be independent from the initial dimension of the assisted firms

[1_example]

How to measure changes inthe outcome variable?

(Ypost- Ypre)

(Ypre)

10

•With pre- post- intervention data, what really matters is separating changes due to the program from changes due to other factors

•What characteristics may be different between treated and non-treated units that put non-treated units at risk of being exposed to different external factors generating changes in the outcome at the same time of the intervention

•Even with discontinuity check with control variables that units around the threshold are indeed similar: this is done even with pure randomized experiments (with small numbers sometimes randomization is unlucky….)

How to choose the control variables

11

“Attrition bias” in outcome data• “Attrition bias” is generated when units with

particularly bad (or good results) drop out from the data used to construct the outcome variable

•Overlooked issue with strong consequences on impact estimates

•E.g.

enterprise support (balance sheet data contains only corporations: treated firms with bad results may close or may loose their corporate status and drop out from the data)

Intuitively: “attrition bias” = programme impact is estimated only for the treated firms with no bad results)

12

E.g.

•R&D and innovation support: treated firms (can be start-ups, small firms) doing particularly well may be lost in the data because of changes in their company names or corporate status occurring in the growing process

•Enterprise support, Job training and social programmes: treated & non-treated units with particularly good (or bad) results may not be anymore available to answer the questionnaires or interviews used to build the outcome variable

“attrition” bias” can severely compromise CIE

check for possible reasons why units maydrop out from the data or check if characteristics of

respondents are similar to non-respondents

13

• For many programmes (e.g. local economic development, enterprise support, R&D-innovation support) extreme caution is needed in attempting to estimate long-term effects

• In the medium and long-run it is likely that a positive programme impulse spreads (with positive or negative spillovers) also to the non-treated units

• Outcomes from the non-treated units become affected by the intervention and cannot be anymore used to estimate the counterfactual

Long-term program effects & CIE

14

•In principle, every type of policy may produce long-run impacts on macro-economic outcomes measured at regional/country level

Estimating such regional impacts “spill-over effects” is to be avoided when the economic importance of the activities produced by the

policy is negligible compared to size of the regional economy and to

the importance of the large number of socio-economic events (unrelated to the program

intervention) that do affect the macro regional/country outcomes

Regional macro-effects & CIE

15

•When the economic importance of the activities produced by the programme is not negligible compared to size of the regional economy proceed as follows:

I) Use rigorous CIE the proximate impacts of the program intervention on firm-level (e.g. how much of the subsidized investment would have been done anyway?)

II) Include the results of CIE in macro-economic regional simulation models yielding multiplying effect for the regional economy

Keep in mind: in the absence of CIE the multipliers used by regional simulations would be applied directly to measures of programme

activity (over-estimating the macro-regional impact)

16

•With similar programmes available at pre-intervention times, pre-intervention data may have been affected by early treatments

•Caution in using pre-intervention characteristics as control variables if you do not include in the analysis the early rounds of the treatment

•Make efforts in gathering data to detect which units were treated also in pre-intervention times

Continuing programmes

17

Practical advantages of PSM•Post-intervention outcome data are not always readily available (delays in releasing recent data by statistical

offices or need of collecting data through questionnaires, interviews, direct

observation….)

•Estimation of PS requires exclusively pre-intervention data on observable characteristics

•PSM reduces the number of non-treated unit in which post-intervention outcome data has to be collected

18

•Designing CIE starts from understanding the rationale of the programme intervention

(….. what is the market imperfection/negative externality that the policy wants to correct)

•E.g. Enterprise support policies:

A) Aimed at producing firm-level investment, that would never take place in the absence of the program intervention (correction of credit market imperfections)

How to choose the outcome variable of the evaluation

19

B) Aimed at boosting business activities in distressed areas

•Rationale: market forces produce non-optimal allocation of economic development with negative externalities (e.g. urban sprawl, traffic congestion, pollution, abandoned areas that may be conducive to crime…..)

•Goal: modifying the geographic allocation of firm-level investment (e.g. new investments wanted in Region A, not in Region B)

C) Aimed at boosting business activities in times with economic crises (Countercyclical policies)

•Goal: modifying the temporal allocation of firm-level investments (firms activities wanted in time I, not in time II)

20

FIRM X (treated) FIRM Y (Non-treated)

Time

Region A(disadvantaged)

Region B

+1 Million € investment

+1 Million € investment

Region B

Example I

Region A(disadvantaged)

21

•For policies A) [aimed at correcting credit market imperfections] the outcome variable has to keep tracks of all firm-activities (e.g. investments) recorded anywhere:

Example I)= zero impact

•For policies B) [targeting distressed areas] the outcome variable has to keep track of firm-activities recorded solely in region A (disadvantaged)

Example I)= positive impact

•For policies C) [countercyclical interventions] the outcome variable has to keep track of firm-activities recorded solely in time I

Example I= uncertain( impact depends on the timing of the investments, not location)

22

Example II

FIRM X (Assisted) FIRM Y (Non-Assisted)

Time

Period I

Period II

+1Million € investment

+1 Million € investment

23

•For policies A) [aimed at correcting credit market imperfections] the outcome variable has to keep tracks of all firm-activities recorded at any time:

Example II= Zero impact

•For policies B) [targeting distressed areas] the outcome variable has to keep track of firm-activities recorded solely in region A (disadvantaged)

Example II)= uncertain (impact depends on location of the investment not time)

•For policies C) [countercyclical interventions] the outcome variable has to keep track of firm-activities recorded solely in time I

Example II= positive impact

24

• If surveys are planned to cover both treated and non treated units, all CIE methods can be applied on survey outcome data

•PSM can be used to generate surveys outcome data:

Surveys are to be run on:all treated units + the matched non-treated

units

•Radius matching procedures are preferable to ensure to have a representative sample of comparable non-treated units

CIE with survey outcome data