IMPRINT Developer’s Workshop December 6-7, 2005
Meta-analytic Reviews of the Effects of Temperature and Vibration on Performance
J.L. Szalma & G. Conway
University of Central Florida
IMPRINT Developer’s Workshop December 6-7, 2005
Performance under Stress
• Stress Degrades Performance
• Relation of Stress to IMPRINT: predict the degree of performance degradation under specific forms of stress
• Currently estimates may not be representative of the entire literature
• Estimates may not reflect interactions with other variables
IMPRINT Developer’s Workshop December 6-7, 2005
Performance under Stress
• Goal for Current work: Based on a quantitative review of the literature, derive estimations of the effect of specific stressors on human performance
• Outcome: Improve IMPRINT as a performance prediction tool by increasing the accuracy of the stress algorithms
IMPRINT Developer’s Workshop December 6-7, 2005
Why Use Meta-Analysis?
• Advantages
• Quantitative synthesis
• Controls for sampling error and low power
• Permits examination potential moderator variables
• Disadvantages
• Generally have to discard many published studies because they do not meet inclusion criteria
• Analysis is only as good as the data that go into it
IMPRINT Developer’s Workshop December 6-7, 2005
Significance Tests• Significance tests are useful, but finding a statistically
significant difference depends on the power of the analysis
• Many publications in behavioral science include statistical analyses of relatively low power
• It would be helpful to have a standardized measure of mean differences that indicates the strength of the effect independent of the results of a significance test and that can be compared across different measures
IMPRINT Developer’s Workshop December 6-7, 2005
The Effect Size
Cohen’s d (Hedge’s g): Standardized Mean difference between two conditions
Range: -∞ --------------- 0 --------------- +∞
s
XXd CE
IMPRINT Developer’s Workshop December 6-7, 2005
Magnitude of Effect
Magnitude of Effect (Cohen, 1988)
Large: .80
Medium: .50
Small: .20
These are only meant to be guidelines
The magnitude of ‘large’ and ‘small’ effects can vary by domain
IMPRINT Developer’s Workshop December 6-7, 2005
General Procedures in Meta Analysis
• Literature Search
• Weighted Average Effect Size
• Variance Partitioning
• Moderator Analysis
• Hierarchical Moderator Analysis
• Criteria for inclusion
IMPRINT Developer’s Workshop December 6-7, 2005
Weighted Effect Sizes
Not all effect sizes are equally accurate
Analogy: Accuracy of a sample mean
The sampling error associated with a given effect size can be estimated
To obtain a more precise estimate of the average effect size, each d is weighted by the reciprocal of its error variance (Hedges & Olkin, 1985).
IMPRINT Developer’s Workshop December 6-7, 2005
Weighted Effect SizesThus,
iiw dwd *
2
1
iiw where
IMPRINT Developer’s Workshop December 6-7, 2005
Error varianceVariance of the d’s
Estimate of the ‘true’ variance in the population of effect sizes (random vs. fixed effects)
Variance Partitioning
σ2d = σ2
δ + σ2e
IMPRINT Developer’s Workshop December 6-7, 2005
Heterogeneous Effect Sizes and Moderator Variables
If the true weighted average effect size were constant across environments, then
02
22ed
and
IMPRINT Developer’s Workshop December 6-7, 2005
If
02 Then other variables are likely to moderate the magnitude and/or direction of the effect size associated with the variable of interest (temperature/vibration)
Analogous to interaction effects in ANOVA
Moderator Analyses
IMPRINT Developer’s Workshop December 6-7, 2005
Moderator Variable: Taxon
• Task Category (IMPRINT Taxon)
•Perception/discrimination
• Information Processing/Problem Solving
•Numerical
•Fine Motor Continuous
•Fine Motor Discrete
IMPRINT Developer’s Workshop December 6-7, 2005
Moderator Variable: Dependent Measure
• Speed
• Accuracy
IMPRINT Developer’s Workshop December 6-7, 2005
Moderator Variable: Intensity
• Range of Temperature, Heat (Intensity)
• Less than 85°F ET
• Greater than 85°F ET
• Range of Temperature, Cold (Intensity)
• Less than 49°F ET
• Greater than 49°F ET
IMPRINT Developer’s Workshop December 6-7, 2005
Rise in Core Body Temperature as a Function of Effective Temperature
IMPRINT Developer’s Workshop December 6-7, 2005
Moderator Variable: Duration
• Less than 1 hour exposure
• Between 1 and 2 hours exposure
• Between 2 and 3 hours of exposure
• Greater than 3 hours of exposure
IMPRINT Developer’s Workshop December 6-7, 2005
Hierarchical Moderator Analysis
• Nesting moderator variables within levels of other moderating variables
• Analogous to multiple interactions in ANOVA
• Precision of effect size estimates decreases as one moves down the hierarchy
IMPRINT Developer’s Workshop December 6-7, 2005
Example: Thermal Stress
Global Effect size
HeatCold
Perception Information Processing
Perception Information Processing
The number of available studies decreases as one moves down the hierarchy
IMPRINT Developer’s Workshop December 6-7, 2005
Study Criteria
• Empirical study involve the application of a heat or cold stress.
• A control group at room temperature
• Enough data to convert temperature into WBGT
• WBGT = 0.567(DB) + 0.393(RH) +3.94• WBGT = (ET-13.1)/0.823• WBGT = 0.7(WB) + 0.3(DB)
IMPRINT Developer’s Workshop December 6-7, 2005
Study Criteria
• Whole body exposed to ambient air heat/cold stress
• The study must involve human performance measures
• Enough information obtain effect size statistic
IMPRINT Developer’s Workshop December 6-7, 2005
Meta Analytic Results
• Literature search resulted in 291 papers published between 1925 and 2003.
• Criteria for Inclusion rules resulted 49 papers
• 57 useable primary studies
• 567 effect sizes
• Independence Problem (analogous to ANOVA)
IMPRINT Developer’s Workshop December 6-7, 2005
The Effect of Thermal Stress on Performance
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
Thermal Stress
Eff
ect
siz
e (d
)
Global Heat Cold
IMPRINT Developer’s Workshop December 6-7, 2005
Thermal Effects as a Function of Task Type
-4.5
-3.5
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
4.5
1
Tasks
Eff
ect
Siz
e (d
)
Heat Cold
PIP/PS FMC
P
IP/PS
FMDFMC
NN
FMD
IMPRINT Developer’s Workshop December 6-7, 2005
Thermal Effects as a Function of Speed and Accuracy
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
Dependent Variable
Eff
ect
size
(d
)
Global Accuracy RT
IMPRINT Developer’s Workshop December 6-7, 2005
Effect Size as a Function of Temperature Range: Heat
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Temperature Range
Eff
ect
size
(d
)
Heat (overall)
ET<=85 F
ET>85 F
IMPRINT Developer’s Workshop December 6-7, 2005
Effect Size as a Function of Temperature Range: Cold
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Temperature Range
Eff
ect
size
(d
)
Cold (overall)
ET<=49F
ET>49F
IMPRINT Developer’s Workshop December 6-7, 2005
Effect Size as a Function of Task and Exposure Duration
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1
Duration on Task
Eff
ect
Siz
e (d
)
<1
1<=t<2
2<=t<3
3<=t
IMPRINT Developer’s Workshop December 6-7, 2005
Duration by Intensity Interaction: Heat
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
1
Duration
Eff
ect
Siz
e (d
)
Heat ET<85F Heat ET>85F
time<2hr
time>2 hr
time<2hr
time>2 hr
IMPRINT Developer’s Workshop December 6-7, 2005
Duration by Intensity Interaction: Cold
-6.5
-5.5
-4.5
-3.5
-2.5
-1.5
-0.5
0.5
1.5
2.5
3.5
4.5
5.5
6.5
Duration
Eff
ect
Siz
e (d
)
Cold ET<49F Cold ET>49F
time<2hr
time>2 hr
time<2hrtime>2 hr
IMPRINT Developer’s Workshop December 6-7, 2005
Conclusions and Implications
• Temperature clearly impacts performance
• Effects are different across Taxons
• Task type interacts with other moderating variables
• Implication: Need consider moderating variables (and interactions) when estimating effect of temperature on performance.
Top Related