Experiments and Quasi-Experiments
Embed Size (px)
Transcript of Experiments and Quasi-Experiments
Experiments and Quasi-ExperimentsJosh LernerEmpirical Methods in Corporate Finance
The problemCorporate finance has not traditionally carefully thought about:Reverse causation.Impact of unobserved third factors.Even when acknowledge, often responses are inadequate:E.g., Kaplan-Ruback .
UnderstandableIn education and labor, there are:Lots of decisions.Relatively minor stakes.Desire to appear innovative:At least in education.Larger stakes, greater risk aversion in finance:Hard to get CEOs, I-bankers to agree to randomized IPOs.
ConsequencesLots of papers about correlationRelatively limited number that decisively show causation.As a resultWill focus here on experiments to address the issues.Somewhat different approach.
Strengths of experimentsAllows careful design of choices.True randomization of participants.Limited worries about sample selection and other issues.
Limitations of experimentsShort time frame and little study compared to real-world choice.Are student participants representative?Modest, low-powered financial stakes.Role of human subjects committee:The cancelled pencil orders.
AgendaWill examine variety of approaches:From less to more practical in finance context.
Using Randomization in Development Economics ResearchDuflo, Glennerster, and KremerWorking Paper
Case for experimentationMany things may be different:Legal regime.Economic conditions.Skill level.These are likely to be correlated with treatment.A study of treated and untreated may thus reflect other influences.
Case for experimentation (2)With experiment, have otherwise identical people:Allows one to capture treatment effectWhile minimizing bias associated with selection effect.
Other biasesIn traditional studies, likely to have data snooping issues:Many regressions run, but only a few reported.Playing with controls and sub-samples.Less opportunities with experiments:But still sub-sample issues.FDA bans in clinical trials.
ImplementationIn development, can be done fairly cheaply:Groups are often searching for solutions.Many programs encounter excess demand.In many cases, programs are phased in over timeMore challenging in corporate finance.
Practical issuesChanges are often done in packages:Makes it harder to assess impact of any change.May deviate from perfect experiment:Selection may be partially non-random.Participation may not be universal.Statistical approaches may partially adjust here.Knowledge spillovers to others.
GeneralityExperiments by definition are micro in scope:Overall effects may be different:Shifts in pricing, externalities, etc. Hawthorne effect:Those in experiment may react to being selected, as do those not selected.Specificity of particular context.
Can experiments work in finance?In many cases, no:Size of stakes.Limited number of players.Differences in needs across firms.Risk aversion.Cost of implementation.
Possible settingsPublic programs:More emphasis on evaluation.Angel groups, and other young firm financiers:Mixture of motivations?!Laboratory experiments that capture essence of problem:Next paper is an example.
Is Pay-for-Performance Detrimental to Innovation?Ederer and MansoWorking Paper, 2008
Split view on incentives and innovationNumber of economics papers showing power of incentive schemes:Typically, basic production activity.Experimental methodology.Negative view in psychology literature of impact on creativity.
MethodologyUse HBSs CLER laboratory.379 subjects.60 minute experiment:20 periods.Students paid between $11 and $40, depending on success.
Methodology (2)Make choices regarding running lemonade stand in each period.Choose one of three locations (schoolmost profitable, stadium, and business district), and sugar level, lemon level, color and price.Different optimum for each location.Suggest initially a non-optimal strategy (business district):Participants can fine-tune or radically alter.
Methodology (3)Three compensation schemes:Fixed wage per period.Expect will explore the least.50% of profits throughout.50% of profits in last 10 periods (exploration).Expect will explore more and get closer to optimum.
ResultsMost likely to sell in school if exploration contract.Figure 1.Most likely to explore in early periods if exploration contract.Figure 2.
Searching performanceLook at for those exploring (leave the business district), when stop (return or converge to a narrow band):Longer for exploration contract.Especially for first ten periods.Especially when also if punitive feature of insufficient profits.Table 1.
More resultsBetter record keeping with incentive pay:Figure 3.More profits with exploration contract:Figure 4.
Assessment and concernsSuggests contracts can effect innovation:Truth in both views.But remaining questions:Would real stakes introduce more risk aversion?Do incentives really work in real world like this?Then why are researcher contracts so flat?What about joint production functions?
The Importance of Holdup in Contracting: Evidence from a Field ExperimentIyer and SchoarWorking Paper, 2008
Here, real stakesSend entrepreneurs to pen market in Chennai, India (>100 shops).Ordered either generic or custom pens.Looked at levels of deposit required, as well as reaction to cancellation.Seeks to test theories of hold-up and renegotiation.
MethodologyReal entrepreneurs used, so familiar with bargaining:Control for ethnicity, etc.Push to complete a deal, according to a script (e..g, deposit required).Randomization in order size and other variables.Average about $25.Cancellations done via phone.
Key findings25% larger deposit required for customized pens.If cancel order ex post, more willingness to renegotiate ifLower deposit.Customized pens.
CommentsCloser to real business situation:Developing country setting allows to replicate real interactions.Though pressure to get real bargain for analysis.But may wonder whether more sophisticated parties, bigger stakes, would affect results.
Does Microfinance Really Help the Poor?MorduchWorking Paper
Regression discontinuity designUnits are assigned to conditions based on a cutoff score.The effect is measured as the discontinuity between treatment and control regression lines at the cutoff:It is not the group mean difference.
RD design, no effect10090807060504030201008070605040302010PrePostnullCutoff
RD design with effectCutoff
RD design with effect100908070605040302010080706050403020PreposteffControlgroupregressionlineTreatmentgroupregressionlineCutoffDifferencebetweengroups
Trade-offWhen properly implemented and analyzed, RD yields an unbiased estimate of treatment effect:it is reasonable to assume that in the absence of the treatment, there would be no discontinuities that naturally occurs between the two groups.
Trade-off (2)Statistical power is considerably less than a randomized experiment of the same size.Effects are unbiased only if the functional form of the relationship between the assignment variable and the outcome variable is correctly modeled:E.g., non-linearities.
AssessmentCook, Shaidsh and Wong :Regression discontinuity studies in education do as well as experiments.But other approaches do not:Differences-in-differences.Careful controls.Other studies suggest dramatic advantages to experiments (e.g., of peer effects).
Application to financeIn many cases, seem like appropriate design, e.g.:Securities regulation cut-offs.Subsidies for small firms.Avoidance of many problems illustrated above.
Paper illustrates remaining difficultiesSeek to understand impact of microfinance:In Bangladesh, half-acre cut-off for participation.But rules appear to be broken:See Table 1, Figure 2.
Paper illustrates remaining difficulties (2)As a result, need to divide villages by whether microfinance was available there.Ends up more like a differences-in-differences approach.
LessonsThis approach seems very reasonable:Better fit for corporate finance.But messy reality has to be grappled with.And limited power of tests may be problematic.
Final thoughtsCorporate finance has been surprisingly casual about these issues.Addressing causality issues is quite important.Practical limitations of many approaches used elsewhereBut more can be done!