Chapter 8: Inferences Based on a Single Sample: Tests of Hypotheses Statistics.
Agenda Group Hypotheses Validity of Inferences from Research Inferences and Errors Types of Validity...
-
Upload
leonel-yellin -
Category
Documents
-
view
219 -
download
1
Transcript of Agenda Group Hypotheses Validity of Inferences from Research Inferences and Errors Types of Validity...
Agenda Agenda
Group Hypotheses
Validity of Inferences from Research
Inferences and Errors
Types of Validity
Threats to Validity
Validity Validity
Inferences and Errors
Types of Validity
Threats to Validity
Inferential hazards Inferential hazards
Empirical tests are inherently ambiguous
Results cannot confirm or prove theory
Possible rival explanations persist
Conclusions about causal effects are clouded by a number of design factors
Lack of control over extraneous influences
Processes of measurement
Inferential hazards Inferential hazards
Two fundamental mistakes possible:
We conclude our theory is TRUE when in
truth it is FALSE (Type One Error)
We conclude our theory is FALSE when in
truth it is TRUE (Type Two Error)
Theory Is Actually:
True False
Conclude TheoryIs True
Conclude TheoryIs False
Correct
Correct
Type I Error
Type II Error
Validity Validity
Has to do with the “degree of doubt”
surrounding our inferences
Doubts about whether we are measuring what
we think we are
Doubts about whether we observe a
relationship
Doubts about whether the observed
relationship is causal
Doubts about whether we can generalize from
the relationship
X
Exposure to violent TV
Y
Aggressive Behavior
XViewing eithera boxing filmor sports film
YAdministration of shocks in a “learning” study
External Validity
Construct(Measurement)Validity
Internal Validity
Generalization to Population in the “Real World”Statistical
Conclusion Validity
?
Construct validity Construct validity
All measures imply a theory
All measures contain some error
Random error (“noise”)
Systematic error (“bias”)
How valid is our interpretation of a
measure?
Statistical conclusion validity
Statistical conclusion validity
Observed relationships depend upon:
Random processes
Number of observations
The “noisiness” (unreliability) of our measures
How certain are we that we have observed a
relationship?
Randomness raiseschance of Type II Error
Hypothesis Is Actually:
True False
Accept Hypothesis
Reject Hypothesis
Correct
Correct
Type I Error
Type II Error
Systematic mistakesraise chances of either I or II, depending
Internal validity Internal validity
Concluding that a relationship is causal
requires:
Covariation
Temporal ordering of observed cause and effect
“Non-spuriousness” i.e., elimination of rival
explanations
How certain are we of a cause-effect
relationship?
External validity External validity
Sound generalizations depend upon:
Representativeness of the sample
The observational setting
Representativeness of the processes
observed
How certain are we that generalizations
are warranted?
Threats to validity Threats to validity
Threats are rival explanations
For our measures
For why we did (or didn’t) observe a
relationship
For what caused the relationship
For the meaning of the relationship in
everyday life
Ambiguous Causal DirectionAmbiguous Causal Direction
Lack of established temporal order may render reverse causation plausible Common in correlational studies
1Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
SelectionSelection
Systematic differences may be present in groups selected for comparison Groups must be made comparable
By random assignment (best)
By matching (not as good)
2Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
HistoryHistory
Changes in the environment may occur between measurements Example: Studying a community intervention
3Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
(A) X (health campaign)
(B) No X ( no campaign)
Y (diet)
Y (diet)
Other events could account for changes in Y
MaturationMaturation
People change naturally over time The longer the time between
measurements, the greater the possibility of maturation effects
4Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
Regression artifactsRegression artifacts
Extreme scores tend to “regress” toward the mean on a second test A problem if tests are used to select groups
and then repeated
5Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
First Time Scores
400510560600
...
Second Time Scores
240280310350
...
690730780800
280
350380
500520
490
640700760790
240
AttritionAttrition
People may drop out of studies May produce non-random group
comparisons
6Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
TestingTesting
Repeated measures can affect each other
7Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
(A) Y
(B) Y
Y
Y
Y (diet) Y (diet)
X (campaign)
X
No X
TestingTesting Measures can also interact with the manipulation
to produce effects (AKA “interaction of testing and treatment”)
7Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
(A) Y
(C)
Y
Y
X
X
(B) Y Y No X
(D) Y No X
InstrumentationInstrumentation
Changes in measurement processes can confound interpretation Instruments should be identical over time and across
groups “Ceiling” and “floor” effects may cloud interpretation of
measurements
8Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
Interactions with selectionInteractions with selection
Different groups maturing at different rates
Different groups experience “local history”
Different groups experience unique “ceiling” or “floor” effects on measures
9Threats to Internal ValidityThreats to Internal ValidityThreats to Internal ValidityThreats to Internal Validity
Reactivity to experimental settingReactivity to experimental setting
People may respond, not to the independent variable per se, but to the situation Experimenter demand
1Threats to Construct ValidityThreats to Construct ValidityThreats to Construct ValidityThreats to Construct Validity
Compensatory equalizationCompensatory equalization
Administrative equity can spoil comparisons Relevant when interventions provide
desirable public goods
2Threats to Construct ValidityThreats to Construct ValidityThreats to Construct ValidityThreats to Construct Validity
Compensatory rivalryCompensatory rivalry
Recognition of treatment can cause “control” groups to compensate
3Threats to Construct ValidityThreats to Construct ValidityThreats to Construct ValidityThreats to Construct Validity
Resentful demoralizationResentful demoralization
Recognition of treatment can cause “control” groups to become despondent, “act up,” etc.
4Threats to Construct ValidityThreats to Construct ValidityThreats to Construct ValidityThreats to Construct Validity
Diffusion of treatmentsDiffusion of treatments
Contact between treatment and control groups can spoil comparisons Possible in experimental interventions May be present in some quasi-experimental
comparisons
5Threats to Construct ValidityThreats to Construct ValidityThreats to Construct ValidityThreats to Construct Validity
Interaction of causal relationship with unitsInteraction of causal relationship with units
The effects might only apply to the groups manipulated or observed Volunteers College students
1Threats to External ValidityThreats to External ValidityThreats to External ValidityThreats to External Validity
Interaction of causal relationship with outcomesInteraction of causal relationship with outcomes
The effects might only apply to particular, observed facets of complex phenomena
Fuller picture of effects might lead to different conclusions
2Threats to External ValidityThreats to External ValidityThreats to External ValidityThreats to External Validity
Interaction of causal relationship with settingInteraction of causal relationship with setting
Experimental setting may be complex or artificial
Effects may be limited to a particular time or place
3Threats to External ValidityThreats to External ValidityThreats to External ValidityThreats to External Validity
For Tuesday For Tuesday
Manipulation, observation, and control of Variables
Experiments
Schutt, Ch. 7 on experimental design
Shadish, Cook & Campbell, Ch. 8 on problems
and solutions in conducting experiments