Econometric analysis and counterfactual studies in … · Econometric analysis and counterfactual...
Transcript of Econometric analysis and counterfactual studies in … · Econometric analysis and counterfactual...
Econometric analysis and counterfactualstudies in the context of IA practices
Giulia Santangelo
http://crie.jrc.ec.europa.eu
Centre for Research onImpact Evaluation
DG EMPL - DG JRC
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
CRIEI Centre for Research on Impact Evaluation
I Joint DG EMPL-DG JRC initiative, established in June 2013I Motto
I Support to MS and DG EMPL to set up the necessaryarrangements for carrying out Counterfactual ImpactEvaluations (CIE) of interventions funded by the EuropeanSocial Fund (ESF)
I TeamI Researchers with a background in Economics and StatisticsI Experience in labour economics, theoretical and applied
econometrics, impact evaluation methodsI Why did DG JRC and DG EMPL decide to put money and
invest in this field?... I come back later on this
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 2/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Presentation Outline
1. Talk: messages
2. Macroeconomic models
3. Counterfactual Impact Evaluations - CIE
4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 3/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
The purpose of my talk
I Not really on Ex-ante Impact Assessment (IA)...more onEx-post Impact Evaluation (CIE)
I However, we have a common interest: we want to knowabout the Impact
I Cross breeding of ideas useful
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 4/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Key messages
I Ex-post CIE can be used for Ex-ante IA
I The use of Ex-post CIE for Ex-ante IA impliesI Promotion of a culture of EvaluationI Staff in charge of IA to be knowledgable on CIE methods
I Ex-post CIE must be planned well in advance in theEx-ante IA
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 5/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Our common interest: ImpactI We need to quantify the effect of a policy
I Why? Evidence based policy making needed toI Improve the quality and effectiveness of public policiesI Promote accountability
I Quantitative methodsI Macroeconomic modelsI Impact evaluation based on counterfactuals
I ...Important to make the distinction between Impact andExpected Impact
I CIE is about quantifying the impact per se of a policyEconometric analysis and counterfactual studies in the context of IA practices CIE Training 6/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Presentation Outline
1. Talk: messages
2. Macroeconomic models
3. Counterfactual Impact Evaluations - CIE
4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 7/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Macroeconomic modelsPros
I Holistic representation of an economic systemI Ex ante and ex post evaluations: possible to evaluate the
intended effect of the program at its design stageI Possibility to test various scenarios
ConsI Theoretically driven: modeler’s perspective
I Different models = different conclusions for a given policyI Calibration issue
I Not causal inferenceI Complex for non experts
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 8/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Presentation Outline
1. Talk: messages
2. Macroeconomic models
3. Counterfactual Impact Evaluations - CIE
4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 9/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
CIE: one tool among others
I Impact evaluation: particular type of evaluationI Seek to answer cause-and-effect questions
I One tool among othersI Must be associated with a theory-based impact evaluationI Must be complemented by monitoring/process evaluation
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 10/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Ingredients for a counterfactual-based evaluationI Well defined intervention targeted at a well defined
populationI Some units i are exposed to the policy intervention (treated
unit) others are not (untreated/control units)
Di = 1 if treatedDi = 0 if untreated
I Expected effect: change of the outcome Yi for the units iexposed to the intervention
Yi = Y1i if treatedYi = Y0i otherwise
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 11/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
What your dataset would look like
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 12/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Ingredients for a counterfactual-based evaluation
I The treatment effect for individual i is defined by Y1i − Y0i
I Problem: only one potential outcome is observed whichimplies that we cannot measure Y1i − Y0i
I Consequence: we must build a counterfactual to be able toestimate the impact of the treatment
I Counterfactual: if we observe Y1i in the treatment group, weneed to estimate Y0i , the “value of the outcome variable forindividual i in the treatment group had it been assigned tothe control group instead”
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 13/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Counterfactual-based program evaluation
In Practice
I Treated units: average outcome ave (Y1i )I Control units used for the counterfactual, average outcome
ave(Y0i )
I Impact of the intervention: ave (Y1i ) − ave(Y0i )I Data: it is necessary to have information on treated and
untreated units (administrative data/surveys)
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 14/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Counterfactual-based program evaluation
Pros and Cons
I Measure the causal effect of an intervention: crucial forinformed policy dialogue
I Challenge: identifying a convincing counterfactualI Examines ”the effect of the cause” but does not explain
”the causes of an effect”I After program implementationI Can be only applied to some specific specific
policies/interventions
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 15/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Corollary 1
I Ex-post CIE can be used for Ex-ante IA
I When available, ex-post evaluations of similar or relatedpolicies should support the ex-ante evaluation process, andin particular help identifying the predicted impact(s) of thepolicy proposal under scrutiny (e.g logic of intervention)
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 16/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
ExampleI Evaluation: Do hospitals make people healthier?I The National Health Interview Survey (NHIS) includes the
following 2 questionsI “During the past 12 months, was the respondent a patient
in a hospital overnight?”Can be used to identify recenthospital visitors.
I “Would you say your health in general is excellent (1), verygood (2), good (3), fair (4), poor (5)?”. Answers to thisquestion define health status (1 = excellent health to 5 =poor health)
I Impact of hospitals? ... we might be tempted to comparethe health status of those who have been to the hospital tothe health of those who have not been to the hospital.
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 17/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Results based on a naive counterfactual
Group Sample Size Mean health status Std. ErrorHospital 7774 2.79 0.014No Hospital 90049 2.07 0.003
I The difference in the means is 0.71, a large and highlysignificant contrast in favor of the non-hospitalized.
I Taken at face value, this result suggests that going to thehospital makes people sicker.
I Are just hospitals full of other sick people who might infectus, and dangerous machines and chemicals that mighthurt us?
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 18/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Selection Bias
I Using individuals who were not hospitalized to measurethe counterfactual is wrong
I Selection bias: people who go to the hospital are probablyless healthy to begin with. Moreover, even afterhospitalization, people who have sought medical care arenot as healthy, on average, as those who never gethospitalized in the first place (though they may well bebetter than they otherwise would have been).
I To measure the effect of the hospitalization, we need tofind a good counterfactual, i.e. a proxy for the health statusof those hospitalized, had they not been hospitalized
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 19/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Corollary 2
I Staff need to be trained so as to understand how to readstudies based on CIE and to judge about the ”quality” ofthe counterfactual used for the estimation of the impact
I CIE is essentially based on econometrics/statisticsI Quantitative background needed
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 20/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Why CRIE?
I Coming back to the previous question: Why CRIE?
I DG EMPL’s concern :I Difficult economic times: pressing need for the EU to
demonstrate the achievements of the European SocialFund (ESF) and other financial instruments
I 2014-2020 programming period: common and ESF-specificprovisions: involve a shift toward impact evaluationculture
I CRIE: to develop a capacity in house on CIE and help DGEMPL and MS when it is question of quantifying the impactof an intervention.
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 21/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Presentation Outline
1. Talk: messages
2. Macroeconomic models
3. Counterfactual Impact Evaluations - CIE
4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 22/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
How to construct the counterfactual?
I Experimental approach
I Quasi-experimental methodsI Difference in Differences (DiD)I Regression Discontinuity Design (RDD)I Propensity Score Matching (PSM)I Instrumental Variables (IV)
I Other methods exist
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 23/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Presentation Outline
1. Talk: messages
2. Macroeconomic models
3. Counterfactual Impact Evaluations - CIE
4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 24/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Randomised experiments - the gold standard
I An essential objective of CIE methods is to identifysituations where we can assume that the selection biasdoes not exist or find ways to correct for it
I Experimental approach = gold standard among CIEmethods because the randomisation solves the selectionbias
I Since treatment and control groups are statisticallyequivalent, differences in outcomes between the groupscan be attributed to the intervention.
I Random assignment into treatment from the pool ofexperiment participants ensures that the treatment andcontrol groups are “statistically” equivalent.
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 25/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
A graphical representationY
tt1 t3
treated
controls
Impact
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 26/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Practical problems with experiments
I Ethical/legal issuesRandomised design can be politically unfeasible because itis hard to justify discrimination between individuals whocan benefit and cannot, especially in poor areas.Counter-arguments:
I Uncertainty about the resultsI Long-run benefits for the societyI In the presence of limited resources, random assignment is
fair and transparentI Some methods of randomisation can attenuate these
issues: ”Randomized Promotion”
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 27/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Job search assistance example
I Characteristics of the interventionI Job-search assistance and counseling.I Information is collected for N = 400 unemployedI Sampled individuals are randomly assigned to a treatment
and control group.I Variable of interest Y is the change in income before-after
(continuous).I What we observe
I Yi , the change in log wageI Di , = 1 if in the job-search assistance intervention,
otherwise = 0I Xi = agei , in the range [15,65]
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 28/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Example - perfect randomisation
Yi = α + ρDi + ui
------------------------------------------------------------------------------
Y | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
D | .9044507 .0563235 16.06 0.000 .793722 1.015179
_cons | 2.50767 .0281617 89.05 0.000 2.452305 2.563034
------------------------------------------------------------------------------Econometric analysis and counterfactual studies in the context of IA practices CIE Training 29/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Example - perfect randomisation
H0 : ave(agei |Di = 1) = ave(agei |Di = 0)
t =(ave(agei |Di = 1)− ave(agei |Di = 0)
)/
√√√√( var(agei |Di = 1)
n1+
var(agei |Di = 0)
n0
)= 0.48
|t | < 1.96 implies H0 cannot be rejected (α = 0.05) =⇒ data supports perfectrandomisation
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 30/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Presentation Outline
1. Talk: messages
2. Macroeconomic models
3. Counterfactual Impact Evaluations - CIE
4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 31/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Quasi-experimental methods
I When randomization not feasible, possibility to rely onstatistical methods that attempt to mimic a randomizationwith the objective to solve the selection bias issue
I From now on, participation to the program is not random.i.e., participation is based on some criteria or/and isvoluntary
I As a consequence, treated and controlled units differ alongobservable and unobservable characteristics: selection bias
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 32/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Differences in differences
t1
intervention
(YiXi
)t1
t2
post-intervention
t3 YiXiDi
t3
I Explore the time dimension of the data to define thecounterfactual
I Beneficiaries and non beneficiaries of the program areobserved before and after the implementation of the policy
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 33/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
A graphical representation
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 34/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Estimation
I The difference in the outcome variable before and aftertreatment in the control group is subtracted from thedifference in outcome before and after treatment in thetreated group
Impact = [CD] − [AB]
I Valid if the counterfactual trend is identical for treated andnon treated groups:common trend assumption
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 35/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Example: firm-simulation class
I Description. A 1-year firm-simulation class is implementedon some bachelor students. The outcome variable is ascore on a entrepreneurial attitude and skills standardizedtest.
I Data. We observe the score before and after theintervention for participants and non-participants alongwith some individual characteristics.
I Impact. What is the effect of the intervention on theentrepreneurial skills of participants? Do they increase orremain unchanged?
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 36/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Results
Simple difference-in-differences
Treatment status
D=0; N=232 D=1; N=168 Difference
Time t1 4.58 5.57 0.99Time t3 8.61 22.42 13.81
Change t3 − t1 4.03 16.85 12.82
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 37/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Check the common trend assumption: Graphicalcheck
44.
55
5.5
mea
n of
yt
0 2 4 6 8 10time
Treatment group Control group
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 38/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Regression Discontinuity Design
Eligibility for a program determined by a rule treatmentassignment based on the value of a continuous variable Sicalled the forcing variable : Treatment:
Di = 1 if Si ≤ cDi = 0 if Si > c
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 39/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Without the intervention, linear case
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 40/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
After the intervention
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 41/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Marginal beneficiaries and non beneficiaries
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 42/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Impact in a snapshot
I Eligibility rule: On either sides of c, individuals have verysimilar characteristics, but some are treated and othersare not
I Counterfactual: units above the cut-off who did notparticipate, i.e marginal non beneficiaries
I Effect of the intervention: difference in the averageperformance between marginal beneficiaries andmarginal non beneficiaries
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 43/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Example: Hunt-for-job support for non natives
Characteristics of the intervention
I To tackle unemployment among the non native population,unemployment offices in Spain put in place in 2010hunt-for-job support.
I Only unemployed non-natives with a tertiary education,aged 40 or less can participate to this program.
I The duration of the support is 6 months.
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 44/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Example: Hunt-for-job support for non natives
What we observe
I Yi , the average difference in monthly log income between2012 and 2010 at the individual level
I Di , =1 if in the support program, =0 otherwiseI Si , the forcing variable, i.e. the age of the individualI c = 40
What we want to measure
I Impact of the intervention on Yi
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 45/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Graphic interpretationImpact of the intervention corresponds to the jump in the valueof Yi at c = 40
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 46/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Training intervention: estimation of the impact
Participating in the hunt-for-a job support increases the monthlyincome by 66 percent
Linear regression Number of obs = 400
F( 3, 396) = 56.64
Prob > F = 0.0000
R-squared = 0.3122
Root MSE = .47317
------------------------------------------------------------------------------
| Robust
Y | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
D | .6634251 .0856094 7.75 0.000 .4951194 .8317309
age | .1181823 .0164631 7.18 0.000 .0858164 .1505482
age2 | -.001377 .0001916 -7.19 0.000 -.0017536 -.0010004
_cons | -2.268112 .3747427 -6.05 0.000 -3.004846 -1.531378
------------------------------------------------------------------------------
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 47/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Corollary 3
I Ex-post CIEs need to be planned well in advanceI CIE necessitates specific data arrangements which should
be outlined in the IA
I Advanced planning is required in order to first identify andthen collect data on outcome indicators but also contextualinformation on both the treated and control groups, beforeand after the policy intervention.
I Impact evaluation is also time and resource demanding.This should be clearly spelled out in the IA.
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 48/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Presentation Outline
1. Talk: messages
2. Macroeconomic models
3. Counterfactual Impact Evaluations - CIE
4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 49/50
1. Talk: messages2. Macroeconomic models3. Counterfactual Impact Evaluations - CIE4. A crash introduction to CIE methods
5. Randomised experiments
6. Quasi-experimental methods
7. Conclusion
Conclusion
I If you are interested in assessing the impact of EU policy:CIE are useful, even for ex-ante IA
Econometric analysis and counterfactual studies in the context of IA practices CIE Training 50/50