Meta-Analysis and Strategy Research
description
Transcript of Meta-Analysis and Strategy Research
![Page 1: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/1.jpg)
Meta-Analysis and Strategy Research
Dan R. Dalton
Kelley School of Business
Indiana University
![Page 2: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/2.jpg)
2
A [Very] Brief History of Research Synthesis
• Averaging Correlations?• Combining Significance Levels?• The Narrative Review (aka “Counting”
Review)• Gene Glass (1976) “Invents” Meta-Analysis• Early Critics – “An Exercise in
Mega-Silliness”
![Page 3: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/3.jpg)
3
An Example of Meta-Analysis(Data Are Simulated)
• Research Question: The Extent to which Equity Holdings by CEOs Are Related to Firms’ Financial Performance
• Proposed Moderator: Expected that this Relationship Will be Moderated by the “Maturity of the Firm” (i.e., Firms that Are Five or Less Years Post-IPO vs. Other)
• Studies Available for Meta-Analysis = 30 (10 are not significant, 10 are positive and
significant, 10 are negative and significant)
![Page 4: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/4.jpg)
4
An Example of Meta-Analysis(Data Are Simulated)
• R = Reliability• RR = Range Restriction• M = Moderator (1 = ≤ 5 yrs. Post-IPO; 2 = > 5 yrs. Post-IPO)
r n Ry Rx RRy RRx M
.26
.39
.37
.29
.23
.11
56
225
192
146
70
325
.8
.8
.8
.8
.8
.8
.8
.8
.8
.8
.8
.8
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
![Page 5: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/5.jpg)
5
“r” - A Bivariate Correlation
• “r” vs. “d”• R-square• Deriving “r” from “d,” “t,” “F-score,” “Z,”
“Chi-Square” …• “r” from Incomplete Information
r = Z/sqrt n
if “n” = 120 and Z = 1.96 with “r” unknown
then r = +/ - .179 (i.e., 1.96/10.95)
![Page 6: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/6.jpg)
6
“r” – A Bivariate Correlation, cntd.
• -17 to +17 and Enter What?
• Discard the Study?
• “r” and the Z-transformation?
• “r” and Statistical Significance
• And, a “Surprise” About Multiple Non-Significant Results
![Page 7: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/7.jpg)
7
“r” – A Bivariate Correlationand “n”
• “r” As an Independent Variable, a Dependent Variable, a Control Variable, a Moderating Variable, a Mediating Variable…
• “n” – The Sample Size from which the “r” Was Calculated
• To Weight the Observed Correlation in Order to Calculate the Mean Weighted Correlation Across All of the Studies
• “n” and the Correlation Matrix
![Page 8: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/8.jpg)
8
Ry (Reliability of y); Rx (Reliability of x)
• Constructs vs. Observed Variables
• Strategic Management Meta-Analyses with Ry = 1 and Rx = 1
• Strategic Management Variables Are Not That Good
• The Choice of Ry and Rx Levels Is Counterintuitive – Lower Ry’s and Rx’s Will Improve the “Corrected r”
• Ry and Rx at .8
![Page 9: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/9.jpg)
9
RRy & RRx (Range Restriction of y and x)
• Analytical Issues of Range Restriction Have Become Increasingly Complex
• In Strategic Management – RRy and RRx as Deliberate Selectivity in the Sample
• Strategic Management and “Survival” Issues
![Page 10: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/10.jpg)
10
Moderation in Meta-Analysis
• In Meta-Analysis a “Moderator” Is a Subgroup
• Profligate Testing for ModeratorsCapitalization on ChanceLoss of Statistical Power
• Moderators Need Not Always Be Operationalized as a Dichotomy
![Page 11: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/11.jpg)
11
Meta-Analytic Procedures and Results
PART 1: # of
Correlations
CombinedSample
Size
MeanTrue ScoreCorrelation
Std Dev:Mean True
ScoreCorrelation
EntireSample
30 9,685 -.026 .283
Moderation: ≤ 5 Yrs. from IPO
16 2,032 .417 .048
Moderation: > 5 Yrs.
from IPO14 7,653 -.144 .188
![Page 12: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/12.jpg)
12
Meta-Analytic Procedures and Results
PART 2: MeanTrue Score
Correlation
80%Credibility
Interval
90%Confidence
Interval
% VarianceAttributableTo Artifacts
EntireSample
-.026 - .389 : .336 - .112 : .059 5.74
Moderation:≤ 5 Yrs. from IPO
.417 .354 : .479 .396 : .437 80.49
Moderation:> 5 Yrs.from IPO
-.144 - .386 : .098 -.061 : -.227 7.26
![Page 13: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/13.jpg)
13
Meta-Analytic Results:Some Diagnostics
• The Magnitude of the Mean True Score Correlation• Does the 90% Confidence Interval Include Zero?
Suggests that the Mean True Score Is Not Significant• Does the 80% Credibility Interval (Difference
between Low and High Estimates) Exceed .11? Suggests the Existence of a Moderator
• Does the % Variance Attributable to Artifacts Exceed 75%? Suggests that a Moderator Is Unlikely
• And, If the Tests Had Relied on Different Rx and Ry Values? [ .7 = .48; .8 = .417; .9 = .37 ]
![Page 14: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/14.jpg)
14
Results Summary
• There is no simple relationship ( -.026, ns) between CEO equity holdings and firm financial performance. There is, however, some evidence of the existence of a moderating variable.
• There is evidence of a moderating effect for time since IPO. The relationship between CEO equity holdings and firm financial performance for firms 5 years or less from IPO is .417, a significant relationship. The diagnostics suggest that a further moderating effect of this result is unlikely.
![Page 15: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/15.jpg)
15
Results Summary, cntd.
• The relationship between CEO equity holdings and firm financial performance for firms more than 5 years from the IPO is -.144, a significant relationship. The diagnostics suggest that a further moderating effect of this result is likely.
![Page 16: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/16.jpg)
16
Other Issues in Meta-Analysis
• Fixed vs. Random Effects ModelsRandom Effects Models – Population Parameters
May Vary Across StudiesFixed Effects Models – Population Parameters Are
Invariant• “File Drawer” Problem• Unreported Null Results• “Fail Safe” Approach• The Issue Is Less a Matter of Fail Safe
Algorithms than of Reliance on Too Few Studies
![Page 17: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/17.jpg)
17
Other Issues in Meta-Analysis, cntd.
• Quality of Data
• Outliers Statistical OutliersEntry Error Outliers
• Sensitivity to Outliers
• The General Question of Discarding Data
• Disclosure and Replicability
![Page 18: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/18.jpg)
18
Other Issues in Meta-Analysis, cntd.
• The Independence of Data• Entering Data that Are Clearly Not
Independent• A Random Selection, Pooling, a Weighted
“r”, a Weighted “n”• An Interesting Catch-22• “Clearly Reflect the Same Construct”• Independence of Samples
Constructive Replication
![Page 19: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/19.jpg)
19
General Guidelines for Meta-Analysis
• There is no need to transform the input values of “r”s.
• When it is necessary to impute the value of “r,” set “r” = 0.
• For observed variables, rely on .8 for the reliability of the dependent and independent variables.
• With observed variables, it will rarely be necessary to assign a range restriction score.
![Page 20: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/20.jpg)
20
General Guidelines for Meta-Analysis, cntd.
• Use a conservative 90% confidence interval for the meta-analysis diagnostics (for these data, 95% would be an interval of -.128 to .075, much wider than the -.112 to .059 reported).
• Use a conservative 80% credibility interval for the meta-analysis diagnostics (for these data, the 90% would have been an interval of -.493 to .448, much wider than the -.389 to .336 reported).
• Where the meta-analysis software provides an option, rely on a “Random Effects Model.”
![Page 21: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/21.jpg)
21
General Guidelines for Meta-Analysis, cntd.
• Assuming every effort has been made for an exhaustive search for meta-analysis input data, you need not be concerned about “file drawer” issues
• Neither weight nor exclude data on the basis of the quality of the study. Instead, run two meta-analyses and compare the results for the entire data set and a reduced data set without the troublesome data
![Page 22: Meta-Analysis and Strategy Research](https://reader036.fdocuments.net/reader036/viewer/2022062409/56814ea5550346895dbc4f73/html5/thumbnails/22.jpg)
22
General Guidelines for Meta-Analysis, cntd.
• Only under extremely rare conditions would there be any concerns about the independence of the data; accordingly, there is no need to combine data from separate “r”s in any manner.
• No need to exclude outliers. Instead, run two meta-analyses and compare the results for the entire data set and a data set without the outliers.