Chapter 9 Expanding on Experimental Designs: Repeated Measures and Quasi- Experiments.
Experiments and Quasi-Experiments
44
Experiments and Quasi- Experiments and Quasi- Experiments Experiments Josh Lerner Empirical Methods in Corporate Finance
description
Experiments and Quasi-Experiments. Josh Lerner Empirical Methods in Corporate Finance. The problem. Corporate finance has not traditionally carefully thought about: Reverse causation. Impact of unobserved third factors. Even when acknowledge, often responses are inadequate: - PowerPoint PPT Presentation
Transcript of Experiments and Quasi-Experiments
Experiments and Quasi-Experiments and Quasi-ExperimentsExperiments
Josh LernerEmpirical Methods in
Corporate Finance
The problemThe problem
Corporate finance has not traditionally carefully thought about:Reverse causation.Impact of unobserved third factors.
Even when acknowledge, often responses are inadequate:
E.g., Kaplan-Ruback [1995].
UnderstandableUnderstandable
In education and labor, there are:Lots of decisions.Relatively minor stakes.Desire to appear innovative:
At least in education.
Larger stakes, greater risk aversion in finance:Hard to get CEOs, I-bankers to agree to
randomized IPOs.
ConsequencesConsequences
Lots of papers about correlation…Relatively limited number that decisively
show causation.As a result…
Will focus here on experiments to address the issues.
Somewhat different approach.
Strengths of experimentsStrengths of experiments
Allows careful design of choices.True randomization of participants.Limited worries about sample
selection and other issues.
Limitations of experimentsLimitations of experiments
Short time frame and little study compared to real-world choice.
Are student participants representative?
Modest, low-powered financial stakes.Role of human subjects committee:
The cancelled pencil orders.
AgendaAgenda
Will examine variety of approaches:From less to more practical in finance
context.
Using Randomization in Using Randomization in Development Economics Development Economics
ResearchResearch
Duflo, Glennerster, and KremerWorking Paper
Case for experimentationCase for experimentation
Many things may be different:Legal regime.Economic conditions.Skill level.
These are likely to be correlated with “treatment.”
A study of treated and untreated may thus reflect other influences.
Case for experimentation (2)Case for experimentation (2)
With experiment, have otherwise identical people:Allows one to capture treatment effect…While minimizing bias associated with
selection effect.
Other biasesOther biases
In traditional studies, likely to have data snooping issues:Many regressions run, but only a few
reported.Playing with controls and sub-samples.
Less opportunities with experiments:But still sub-sample issues.
FDA bans in clinical trials.
ImplementationImplementation
In development, can be done fairly cheaply:Groups are often searching for solutions.Many programs encounter excess
demand.In many cases, programs are phased in
over timeMore challenging in corporate finance.
Practical issuesPractical issues
Changes are often done in packages: Makes it harder to assess impact of any change.
May deviate from perfect experiment: Selection may be partially non-random. Participation may not be universal.
Statistical approaches may partially adjust here.
Knowledge spillovers to others.
GeneralityGenerality
Experiments by definition are micro in scope:Overall effects may be different:
Shifts in pricing, externalities, etc. “Hawthorne” effect:
Those in experiment may react to being selected, as do those not selected.
Specificity of particular context.
Can experiments work in Can experiments work in finance?finance?
In many cases, no:Size of stakes.Limited number of players.Differences in needs across firms.Risk aversion.Cost of implementation.
Possible settingsPossible settings
Public programs:More emphasis on evaluation.
Angel groups, and other young firm financiers:Mixture of motivations?!
Laboratory experiments that capture essence of problem:Next paper is an example.
Is Pay-for-Performance Is Pay-for-Performance Detrimental to Innovation?Detrimental to Innovation?
Ederer and MansoWorking Paper, 2008
Split view on incentives and Split view on incentives and innovationinnovation
Number of economics papers showing power of incentive schemes:Typically, basic production activity.Experimental methodology.
Negative view in psychology literature of impact on creativity.
MethodologyMethodology
Use HBS’s CLER laboratory.379 subjects.60 minute experiment:
20 periods.Students paid between $11 and $40,
depending on success.
Methodology (2)Methodology (2)
Make choices regarding running lemonade stand in each period.
Choose one of three locations (school—most profitable, stadium, and business district), and sugar level, lemon level, color and price.
Different optimum for each location. Suggest initially a non-optimal strategy
(business district): Participants can fine-tune or radically alter.
Methodology (3)Methodology (3)
Three compensation schemes:Fixed wage per period.
Expect will explore the least.50% of profits throughout.50% of profits in last 10 periods
(exploration).Expect will explore more and get closer to
optimum.
ResultsResults
Most likely to sell in school if exploration contract.Figure 1.
Most likely to explore in early periods if exploration contract.Figure 2.
Searching performanceSearching performance
Look at for those exploring (leave the business district), when stop (return or converge to a narrow band):Longer for exploration contract.
Especially for first ten periods.Especially when also if punitive feature of
insufficient profits. Table 1.
More resultsMore results
Better record keeping with incentive pay:Figure 3.
More profits with exploration contract:Figure 4.
Assessment and concernsAssessment and concerns
Suggests contracts can effect innovation:Truth in both views.
But remaining questions:Would real stakes introduce more risk
aversion?Do incentives really work in real world like
this?Then why are researcher contracts so flat?
What about joint production functions?
The Importance of Holdup in The Importance of Holdup in Contracting: Evidence from Contracting: Evidence from
a Field Experimenta Field Experiment
Iyer and SchoarWorking Paper, 2008
Here, real stakesHere, real stakes
Send entrepreneurs to pen market in Chennai, India (>100 shops).
Ordered either generic or custom pens.
Looked at levels of deposit required, as well as reaction to cancellation.Seeks to test theories of hold-up and
renegotiation.
MethodologyMethodology
Real entrepreneurs used, so familiar with bargaining:Control for ethnicity, etc.Push to complete a deal, according to a
script (e..g, deposit required).Randomization in order size and other
variables.Average about $25.
Cancellations done via phone.
Key findingsKey findings
25% larger deposit required for customized pens.
If cancel order ex post, more willingness to renegotiate if…Lower deposit.Customized pens.
CommentsComments
Closer to real business situation:Developing country setting allows to
replicate real interactions.Though pressure to get real bargain for
analysis.But may wonder whether more
sophisticated parties, bigger stakes, would affect results.
Does Microfinance Really Help Does Microfinance Really Help the Poor?the Poor?
MorduchWorking Paper
Regression discontinuity designRegression discontinuity design
Units are assigned to conditions based on a cutoff score.
The effect is measured as the discontinuity between treatment and control regression lines at the cutoff: It is not the group mean difference.
RD design, no effectRD design, no effect
1009080706050403020100
80
70
60
50
40
30
20
10
Pre
Postnull
Cutoff
RD design with effectRD design with effect
1009080706050403020100
80
70
60
50
40
30
20
Pre
Posteff
Cutoff
RD design with effectRD design with effect
1009080706050403020100
80
70
60
50
40
30
20
Pre
posteff
Controlgroup
regressionline
Treatmentgroup
regressionline
Cutoff
Differencebetweengroups
Trade-offTrade-off
When properly implemented and analyzed, RD yields an unbiased estimate of treatment effect:it is reasonable to assume that in the
absence of the treatment, there would be no discontinuities that naturally occurs between the two groups.
Trade-off (2)Trade-off (2)
Statistical power is considerably less than a randomized experiment of the same size.
Effects are unbiased only if the functional form of the relationship between the assignment variable and the outcome variable is correctly modeled: E.g., non-linearities.
AssessmentAssessment
Cook, Shaidsh and Wong [2006]: Regression discontinuity studies in education do
as well as experiments. But other approaches do not:
Differences-in-differences. Careful controls.
Other studies suggest dramatic advantages to experiments (e.g., of peer effects).
Application to financeApplication to finance
In many cases, seem like appropriate design, e.g.:Securities regulation cut-offs.Subsidies for small firms.
Avoidance of many problems illustrated above.
Paper illustrates remaining Paper illustrates remaining difficultiesdifficulties
Seek to understand impact of microfinance:
In Bangladesh, half-acre cut-off for participation.
But rules appear to be broken:See Table 1, Figure 2.
Paper illustrates remaining Paper illustrates remaining difficulties (2)difficulties (2)
As a result, need to divide villages by whether microfinance was available there.
Ends up more like a differences-in-differences approach.
LessonsLessons
This approach seems very reasonable:Better fit for corporate finance.But messy reality has to be grappled
with.And limited power of tests may be
problematic.
Final thoughtsFinal thoughts
Corporate finance has been surprisingly casual about these issues.
Addressing causality issues is quite important.
Practical limitations of many approaches used elsewhere…But more can be done!
