Econometric analysis and counterfactual studies in … · Econometric analysis and counterfactual...

50
Econometric analysis and counterfactual studies in the context of IA practices Giulia Santangelo http://crie.jrc.ec.europa.eu Centre for Research on Impact Evaluation DG EMPL - DG JRC

Transcript of Econometric analysis and counterfactual studies in … · Econometric analysis and counterfactual...

Econometric analysis and counterfactualstudies in the context of IA practices

Giulia Santangelo

http://crie.jrc.ec.europa.eu

Centre for Research onImpact Evaluation

DG EMPL - DG JRC

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

CRIEI Centre for Research on Impact Evaluation

I Joint DG EMPL-DG JRC initiative, established in June 2013I Motto

I Support to MS and DG EMPL to set up the necessaryarrangements for carrying out Counterfactual ImpactEvaluations (CIE) of interventions funded by the EuropeanSocial Fund (ESF)

I TeamI Researchers with a background in Economics and StatisticsI Experience in labour economics, theoretical and applied

econometrics, impact evaluation methodsI Why did DG JRC and DG EMPL decide to put money and

invest in this field?... I come back later on this

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 2/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Presentation Outline

1. Talk: messages

2. Macroeconomic models

3. Counterfactual Impact Evaluations - CIE

4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 3/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

The purpose of my talk

I Not really on Ex-ante Impact Assessment (IA)...more onEx-post Impact Evaluation (CIE)

I However, we have a common interest: we want to knowabout the Impact

I Cross breeding of ideas useful

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 4/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Key messages

I Ex-post CIE can be used for Ex-ante IA

I The use of Ex-post CIE for Ex-ante IA impliesI Promotion of a culture of EvaluationI Staff in charge of IA to be knowledgable on CIE methods

I Ex-post CIE must be planned well in advance in theEx-ante IA

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 5/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Our common interest: ImpactI We need to quantify the effect of a policy

I Why? Evidence based policy making needed toI Improve the quality and effectiveness of public policiesI Promote accountability

I Quantitative methodsI Macroeconomic modelsI Impact evaluation based on counterfactuals

I ...Important to make the distinction between Impact andExpected Impact

I CIE is about quantifying the impact per se of a policyEconometric analysis and counterfactual studies in the context of IA practices CIE Training 6/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Presentation Outline

1. Talk: messages

2. Macroeconomic models

3. Counterfactual Impact Evaluations - CIE

4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 7/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Macroeconomic modelsPros

I Holistic representation of an economic systemI Ex ante and ex post evaluations: possible to evaluate the

intended effect of the program at its design stageI Possibility to test various scenarios

ConsI Theoretically driven: modeler’s perspective

I Different models = different conclusions for a given policyI Calibration issue

I Not causal inferenceI Complex for non experts

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 8/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Presentation Outline

1. Talk: messages

2. Macroeconomic models

3. Counterfactual Impact Evaluations - CIE

4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 9/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

CIE: one tool among others

I Impact evaluation: particular type of evaluationI Seek to answer cause-and-effect questions

I One tool among othersI Must be associated with a theory-based impact evaluationI Must be complemented by monitoring/process evaluation

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 10/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Ingredients for a counterfactual-based evaluationI Well defined intervention targeted at a well defined

populationI Some units i are exposed to the policy intervention (treated

unit) others are not (untreated/control units)

Di = 1 if treatedDi = 0 if untreated

I Expected effect: change of the outcome Yi for the units iexposed to the intervention

Yi = Y1i if treatedYi = Y0i otherwise

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 11/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

What your dataset would look like

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 12/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Ingredients for a counterfactual-based evaluation

I The treatment effect for individual i is defined by Y1i − Y0i

I Problem: only one potential outcome is observed whichimplies that we cannot measure Y1i − Y0i

I Consequence: we must build a counterfactual to be able toestimate the impact of the treatment

I Counterfactual: if we observe Y1i in the treatment group, weneed to estimate Y0i , the “value of the outcome variable forindividual i in the treatment group had it been assigned tothe control group instead”

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 13/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Counterfactual-based program evaluation

In Practice

I Treated units: average outcome ave (Y1i )I Control units used for the counterfactual, average outcome

ave(Y0i )

I Impact of the intervention: ave (Y1i ) − ave(Y0i )I Data: it is necessary to have information on treated and

untreated units (administrative data/surveys)

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 14/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Counterfactual-based program evaluation

Pros and Cons

I Measure the causal effect of an intervention: crucial forinformed policy dialogue

I Challenge: identifying a convincing counterfactualI Examines ”the effect of the cause” but does not explain

”the causes of an effect”I After program implementationI Can be only applied to some specific specific

policies/interventions

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 15/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Corollary 1

I Ex-post CIE can be used for Ex-ante IA

I When available, ex-post evaluations of similar or relatedpolicies should support the ex-ante evaluation process, andin particular help identifying the predicted impact(s) of thepolicy proposal under scrutiny (e.g logic of intervention)

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 16/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

ExampleI Evaluation: Do hospitals make people healthier?I The National Health Interview Survey (NHIS) includes the

following 2 questionsI “During the past 12 months, was the respondent a patient

in a hospital overnight?”Can be used to identify recenthospital visitors.

I “Would you say your health in general is excellent (1), verygood (2), good (3), fair (4), poor (5)?”. Answers to thisquestion define health status (1 = excellent health to 5 =poor health)

I Impact of hospitals? ... we might be tempted to comparethe health status of those who have been to the hospital tothe health of those who have not been to the hospital.

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 17/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Results based on a naive counterfactual

Group Sample Size Mean health status Std. ErrorHospital 7774 2.79 0.014No Hospital 90049 2.07 0.003

I The difference in the means is 0.71, a large and highlysignificant contrast in favor of the non-hospitalized.

I Taken at face value, this result suggests that going to thehospital makes people sicker.

I Are just hospitals full of other sick people who might infectus, and dangerous machines and chemicals that mighthurt us?

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 18/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Selection Bias

I Using individuals who were not hospitalized to measurethe counterfactual is wrong

I Selection bias: people who go to the hospital are probablyless healthy to begin with. Moreover, even afterhospitalization, people who have sought medical care arenot as healthy, on average, as those who never gethospitalized in the first place (though they may well bebetter than they otherwise would have been).

I To measure the effect of the hospitalization, we need tofind a good counterfactual, i.e. a proxy for the health statusof those hospitalized, had they not been hospitalized

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 19/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Corollary 2

I Staff need to be trained so as to understand how to readstudies based on CIE and to judge about the ”quality” ofthe counterfactual used for the estimation of the impact

I CIE is essentially based on econometrics/statisticsI Quantitative background needed

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 20/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Why CRIE?

I Coming back to the previous question: Why CRIE?

I DG EMPL’s concern :I Difficult economic times: pressing need for the EU to

demonstrate the achievements of the European SocialFund (ESF) and other financial instruments

I 2014-2020 programming period: common and ESF-specificprovisions: involve a shift toward impact evaluationculture

I CRIE: to develop a capacity in house on CIE and help DGEMPL and MS when it is question of quantifying the impactof an intervention.

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 21/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Presentation Outline

1. Talk: messages

2. Macroeconomic models

3. Counterfactual Impact Evaluations - CIE

4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 22/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

How to construct the counterfactual?

I Experimental approach

I Quasi-experimental methodsI Difference in Differences (DiD)I Regression Discontinuity Design (RDD)I Propensity Score Matching (PSM)I Instrumental Variables (IV)

I Other methods exist

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 23/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Presentation Outline

1. Talk: messages

2. Macroeconomic models

3. Counterfactual Impact Evaluations - CIE

4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 24/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Randomised experiments - the gold standard

I An essential objective of CIE methods is to identifysituations where we can assume that the selection biasdoes not exist or find ways to correct for it

I Experimental approach = gold standard among CIEmethods because the randomisation solves the selectionbias

I Since treatment and control groups are statisticallyequivalent, differences in outcomes between the groupscan be attributed to the intervention.

I Random assignment into treatment from the pool ofexperiment participants ensures that the treatment andcontrol groups are “statistically” equivalent.

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 25/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

A graphical representationY

tt1 t3

treated

controls

Impact

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 26/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Practical problems with experiments

I Ethical/legal issuesRandomised design can be politically unfeasible because itis hard to justify discrimination between individuals whocan benefit and cannot, especially in poor areas.Counter-arguments:

I Uncertainty about the resultsI Long-run benefits for the societyI In the presence of limited resources, random assignment is

fair and transparentI Some methods of randomisation can attenuate these

issues: ”Randomized Promotion”

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 27/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Job search assistance example

I Characteristics of the interventionI Job-search assistance and counseling.I Information is collected for N = 400 unemployedI Sampled individuals are randomly assigned to a treatment

and control group.I Variable of interest Y is the change in income before-after

(continuous).I What we observe

I Yi , the change in log wageI Di , = 1 if in the job-search assistance intervention,

otherwise = 0I Xi = agei , in the range [15,65]

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 28/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Example - perfect randomisation

Yi = α + ρDi + ui

------------------------------------------------------------------------------

Y | Coef. Std. Err. t P>|t| [95% Conf. Interval]

-------------+----------------------------------------------------------------

D | .9044507 .0563235 16.06 0.000 .793722 1.015179

_cons | 2.50767 .0281617 89.05 0.000 2.452305 2.563034

------------------------------------------------------------------------------Econometric analysis and counterfactual studies in the context of IA practices CIE Training 29/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Example - perfect randomisation

H0 : ave(agei |Di = 1) = ave(agei |Di = 0)

t =(ave(agei |Di = 1)− ave(agei |Di = 0)

)/

√√√√( var(agei |Di = 1)

n1+

var(agei |Di = 0)

n0

)= 0.48

|t | < 1.96 implies H0 cannot be rejected (α = 0.05) =⇒ data supports perfectrandomisation

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 30/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Presentation Outline

1. Talk: messages

2. Macroeconomic models

3. Counterfactual Impact Evaluations - CIE

4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 31/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Quasi-experimental methods

I When randomization not feasible, possibility to rely onstatistical methods that attempt to mimic a randomizationwith the objective to solve the selection bias issue

I From now on, participation to the program is not random.i.e., participation is based on some criteria or/and isvoluntary

I As a consequence, treated and controlled units differ alongobservable and unobservable characteristics: selection bias

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 32/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Differences in differences

t1

intervention

(YiXi

)t1

t2

post-intervention

t3 YiXiDi

t3

I Explore the time dimension of the data to define thecounterfactual

I Beneficiaries and non beneficiaries of the program areobserved before and after the implementation of the policy

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 33/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

A graphical representation

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 34/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Estimation

I The difference in the outcome variable before and aftertreatment in the control group is subtracted from thedifference in outcome before and after treatment in thetreated group

Impact = [CD] − [AB]

I Valid if the counterfactual trend is identical for treated andnon treated groups:common trend assumption

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 35/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Example: firm-simulation class

I Description. A 1-year firm-simulation class is implementedon some bachelor students. The outcome variable is ascore on a entrepreneurial attitude and skills standardizedtest.

I Data. We observe the score before and after theintervention for participants and non-participants alongwith some individual characteristics.

I Impact. What is the effect of the intervention on theentrepreneurial skills of participants? Do they increase orremain unchanged?

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 36/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Results

Simple difference-in-differences

Treatment status

D=0; N=232 D=1; N=168 Difference

Time t1 4.58 5.57 0.99Time t3 8.61 22.42 13.81

Change t3 − t1 4.03 16.85 12.82

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 37/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Check the common trend assumption: Graphicalcheck

44.

55

5.5

mea

n of

yt

0 2 4 6 8 10time

Treatment group Control group

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 38/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Regression Discontinuity Design

Eligibility for a program determined by a rule treatmentassignment based on the value of a continuous variable Sicalled the forcing variable : Treatment:

Di = 1 if Si ≤ cDi = 0 if Si > c

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 39/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Without the intervention, linear case

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 40/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

After the intervention

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 41/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Marginal beneficiaries and non beneficiaries

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 42/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Impact in a snapshot

I Eligibility rule: On either sides of c, individuals have verysimilar characteristics, but some are treated and othersare not

I Counterfactual: units above the cut-off who did notparticipate, i.e marginal non beneficiaries

I Effect of the intervention: difference in the averageperformance between marginal beneficiaries andmarginal non beneficiaries

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 43/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Example: Hunt-for-job support for non natives

Characteristics of the intervention

I To tackle unemployment among the non native population,unemployment offices in Spain put in place in 2010hunt-for-job support.

I Only unemployed non-natives with a tertiary education,aged 40 or less can participate to this program.

I The duration of the support is 6 months.

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 44/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Example: Hunt-for-job support for non natives

What we observe

I Yi , the average difference in monthly log income between2012 and 2010 at the individual level

I Di , =1 if in the support program, =0 otherwiseI Si , the forcing variable, i.e. the age of the individualI c = 40

What we want to measure

I Impact of the intervention on Yi

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 45/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Graphic interpretationImpact of the intervention corresponds to the jump in the valueof Yi at c = 40

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 46/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Training intervention: estimation of the impact

Participating in the hunt-for-a job support increases the monthlyincome by 66 percent

Linear regression Number of obs = 400

F( 3, 396) = 56.64

Prob > F = 0.0000

R-squared = 0.3122

Root MSE = .47317

------------------------------------------------------------------------------

| Robust

Y | Coef. Std. Err. t P>|t| [95% Conf. Interval]

-------------+----------------------------------------------------------------

D | .6634251 .0856094 7.75 0.000 .4951194 .8317309

age | .1181823 .0164631 7.18 0.000 .0858164 .1505482

age2 | -.001377 .0001916 -7.19 0.000 -.0017536 -.0010004

_cons | -2.268112 .3747427 -6.05 0.000 -3.004846 -1.531378

------------------------------------------------------------------------------

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 47/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Corollary 3

I Ex-post CIEs need to be planned well in advanceI CIE necessitates specific data arrangements which should

be outlined in the IA

I Advanced planning is required in order to first identify andthen collect data on outcome indicators but also contextualinformation on both the treated and control groups, beforeand after the policy intervention.

I Impact evaluation is also time and resource demanding.This should be clearly spelled out in the IA.

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 48/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Presentation Outline

1. Talk: messages

2. Macroeconomic models

3. Counterfactual Impact Evaluations - CIE

4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 49/50

1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods

5. Randomised experiments

6. Quasi-experimental methods

7. Conclusion

Conclusion

I If you are interested in assessing the impact of EU policy:CIE are useful, even for ex-ante IA

Econometric analysis and counterfactual studies in the context of IA practices CIE Training 50/50