Chance Models, Hypothesis Testing, Power Q560: Experimental Methods in Cognitive Science Lecture 6.
Lecture 9: Experimental Methods
Transcript of Lecture 9: Experimental Methods
![Page 1: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/1.jpg)
Lecture 9: Experimental Methods
![Page 2: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/2.jpg)
Research Design
Gale M. Lucas, PhD
…or “How to run a study”
![Page 3: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/3.jpg)
Schacter & Singer• Claim we don't automatically know when we are
happy, angry. Instead, we label our emotions by considering situational cues. This labeling process depends on two factors:– something in world triggers general, nonspecific
arousal marked by increased heart rate, tightening of the stomach, and rapid breathing.
– people examine perceptual cues and their thoughts about the situation for clues for what has caused the emotion.
• Essentially 2 factor model– Appraisal a purely cognitive process that provides
context for interpreting body state
![Page 4: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/4.jpg)
Overview of “how to run a study”
• Choose your variables, make them concrete
– IV, DV
• Choose your method
– Correlational or experimental
• Analyze data
– Depends on “type” of measurement
![Page 5: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/5.jpg)
Choose your variables, make
them concrete• What do you want to study?
• Variables (or constructs)
• Operational definitions
– A clear, measurable definition of a construct
![Page 6: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/6.jpg)
Our example
• Belonging is a fundamental need, like hunger
• But people feel need to different degrees…
Construct: Need to belong (NTB)
![Page 7: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/7.jpg)
Our example
• Operational definition: Need to Belong
(NTB) Scale (Leary et al., 2001)
• High scores on items 2, 4, 5, 6, 8, 9, 10
• Low scores on items 1, 3, 7
![Page 8: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/8.jpg)
Practice
• Come up with operational definitions for each of the following:
– Trust
– Aggression
– Depression
![Page 9: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/9.jpg)
Choose your variables, make
them concrete• What do you want to study?
• Variables (or constructs)
• Operational definitions
– A clear, measurable definition of a construct
– Specify operational definition for IV and DV
![Page 10: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/10.jpg)
IVs and DVs
• IV: Independent Variable (“manipulated”)
– Variable that you think causes the effect on
another variable
– Variable that is manipulated in an experiment
• DV: Dependent Variable (“measured”)
– The outcome variable (you think it has effect on)
– Variable measured in an experiment
Warning: some
foreshadowing
![Page 11: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/11.jpg)
Overview of “how to run a study”
• Choose your variables, make them concrete
– IV, DV
• Choose your method
– Correlational or experimental
– Depends on your question
?
![Page 12: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/12.jpg)
Our example
• What would that hunger make us do?
– And socially sensitive/aware
– Could ask two questions here
![Page 13: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/13.jpg)
Our example
• “Is NTB associated with social sensitivity?”
• “Does NTB cause heightened sensitivity?”
Correlational
Experimental
NTB Sensitivity
NTB Sensitivity
![Page 14: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/14.jpg)
Correlational vs experimental
• Recall, there was something different
about IVs and DVs in experimental…?
• Correlational
– Measure IV and measure DV
– Our example?
• Experimental
– Manipulate IV and measure DV
– Our example?
?
![Page 15: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/15.jpg)
Is NTB associated with sensitivity?
• What kind of research tests this question?
• Correlational research
• Picket, Gardner, and Knowles
– NTB scale
– DANVA: Number of facial expressions correctly
identified
r = .38What does that mean?
![Page 16: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/16.jpg)
Correlation Coefficient -- r
Correlation
coefficient
Indicates direction
of relationship
(positive or negative)
Indicates strength
of relationship
(0.00 to 1.00)
r = 0.63+r = +0.50
![Page 17: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/17.jpg)
Problems with correlational research
• Only shows there’s an association
• We can’t say NTB causes social sensitivity,
why not…?
• It could be some other variable causing both
– Third variable problem
• Sensitivity could cause NTB
– Directionality problem
NTB Sensitivity
NTB Sensitivity
Third Variable
![Page 18: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/18.jpg)
Problems with correlational research
• Third variable problem
NTBSocial
sensitivity
Motivation
Please
experimenter
Else?
![Page 19: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/19.jpg)
Problems with correlational research
• Directionality problem
NTB Social
sensitivity
![Page 20: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/20.jpg)
Problems with correlational research
• …and there will always another third variable
that you didn’t think of!
• So, conduct experimental research!
– What makes an experiment? What do you have to
do?
– Random assignment to conditions that only
differ on (thus manipulate) one variable
• Our example?Internal
validity!
![Page 21: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/21.jpg)
Experimental NTB research
• Picket, Gardner, and Knowles
– What did they have to do here?
– Manipulate belonging needs
• Measure social sensitivity
![Page 22: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/22.jpg)
Experimental NTB research
• NTB causes an increase in social sensitivity!
0
20
40
60
80
100
120
Control Rejection
So
cia
l s
en
sit
ivit
y
![Page 23: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/23.jpg)
Benefit of experimental research
• Experimental research wins!
• Right?
• Do we only care about outcomes these
kinds of manipulations can effect?
• What important outcomes could NTB
effect? Think big! How might hunger affect
us?
![Page 24: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/24.jpg)
Benefits of correlational research
• Real world outcomes
that really matter
0
0.05
0.1
0.15
0.2
Low High
Males
Females
Age-a
dju
ste
d m
ort
alit
y r
ate
Belonging
External
validity!
![Page 25: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/25.jpg)
Problems with experimental research
• Hard to make manipulations powerful enough
to impact these kinds of outcomes
• Would you really want to?
– Ethical operationalizations?
– Ethically affect those outcomes?
• Even if didn’t have ethics….
– Can’t practically manipulate some things
![Page 26: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/26.jpg)
Overview of “how to run a study”
• Choose your variables, make them concrete
– IV, DV
• Choose your method
– Correlational or experimental
• Analyze data
– Depends on “type” of measurement
![Page 27: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/27.jpg)
Our example
• Please access the link that was emailed to
you and complete the study…
– ---once you are done, you can get up and stretch
• What did you experience?
• What kind of study was this?
• What am I testing? And why these 3 groups?
![Page 28: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/28.jpg)
Data has been collected…
From the raw data it is impossible to tell if the groups differ!
Population
of
students
Sample
Control Life alone
4 5 6 1 3 6
2 7 7 3 2 7
4 6 7 2 5 5
2 3 7 5 7 3
7 6 7 2 3 7
6 1 7 4 2 5
3 1 2 2 2 4
1 1 3 4 2 5
6 1 4 6 5 1
2 7 7 2 7 2
2 7 2 7 1 3
4 7 6 7 5 4
7 2 4 6 4 7
3 6 5 6 3 4
1 7 7 3 5 5
Accident prone
![Page 29: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/29.jpg)
…so run a statistical test
• Which one?
• It depends. …. On whether you’ve run a
correlational or experimental study, right?
– Calculate a correlation coefficient, r, if we ran
a correlational study, right?
– WRONG!
• It depends… on “type” of IV and DV
![Page 30: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/30.jpg)
Overview of “how to run a study”
• Choose your variables, make them concrete
– IV, DV
• Choose your method
– Correlational or experimental
• Analyze data
– Depends on “type” of measurement
![Page 31: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/31.jpg)
“Type” of IVs and DVs
• Both IV and DV can either be “categorical”
or “continuous”
– “groups” vs “how much?”
• Think continuous, think “continuum”
___________________
1 2 3 4 5 6 7
– Ask yourself: “can I put these in order?”
![Page 32: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/32.jpg)
Continuous IV
(e.g., how much)
Categorical IV
(e.g, exp vs control)
Continuous
DV (e.g.,
how much)
Categorical
DV (e.g.,
yes/no)
![Page 33: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/33.jpg)
Continuous IV
(e.g., how much)
Categorical IV
(e.g, exp vs control)
Continuous
DV (e.g.,
how much)t-test
Categorical
DV (e.g.,
yes/no)
![Page 34: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/34.jpg)
Continuous IV
(e.g., how much)
Categorical IV
(e.g, exp vs control)
Continuous
DV (e.g.,
how much)
correlation t-test
Categorical
DV (e.g.,
yes/no)
![Page 35: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/35.jpg)
Continuous IV
(e.g., how much)
Categorical IV
(e.g, exp vs control)
Continuous
DV (e.g.,
how much)
correlation t-test or ANOVA
Categorical
DV (e.g.,
yes/no)
![Page 36: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/36.jpg)
Continuous IV
(e.g., how much)
Categorical IV
(e.g, exp vs control)
Continuous
DV (e.g.,
how much)
correlation or
regressiont-test or ANOVA
Categorical
DV (e.g.,
yes/no)
![Page 37: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/37.jpg)
Continuous IV
(e.g., how much)
Categorical IV
(e.g, exp vs control)
Continuous
DV (e.g.,
how much)
correlation or
regressiont-test or ANOVA
Categorical
DV (e.g.,
yes/no)
chi-square test
![Page 38: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/38.jpg)
Continuous IV
(e.g., how much)
Categorical IV
(e.g, exp vs control)
Continuous
DV (e.g.,
how much)
correlation or
regressiont-test or ANOVA
Categorical
DV (e.g.,
yes/no)
logistical
regression
chi-square test
![Page 39: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/39.jpg)
What test would you use?
• Let’s go back:
– Recall Picket, Gardner, and Knowles’ r = .38
– Was that correct? Why?
– What test here?
0
20
40
60
80
100
120
Control Rejection
So
cia
l s
en
sit
ivit
y
![Page 40: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/40.jpg)
What test would you use?
• Practice question:
– I ran a study to test whether people with high
need to belong smile more
– What do you need to know to figure out what
test I should use?
– IV: NTB scores
– DV: count of smiles during 5 minute
conversation
– If you had high vs low NTB?
– If you had high, med, low NTB?
– Are these last 2 experimental or correlational?
![Page 41: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/41.jpg)
What test would you use?
• Practice question:
– I ran a study to test whether people with high
need to belong smile more
– What do you need to know to figure out what
test I should use?
– IV: NTB scores
– DV: smile or not in photo
– If you had high vs low NTB?
![Page 42: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/42.jpg)
What test would you use?
• …to analyze our data?• ---while I get the data ready, practice & then take a break
• Let’s run that test!
• ...what if I had only randomly assigned you to
“live alone” vs “acceptance” conditions?
• …what if I had you provide your NTB scores
instead of assigning you to conditions?
![Page 43: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/43.jpg)
Could we publish our data?
• Write down: at least one reason why we
could NOT publish our results?
• Lacking IRB approval & informed consent
• Analyze each item separately increases
chance of “false positive”
• Concerns about our sample
– Too small “N” (sample size)
– Findings may not generalize
– Demand characteristics
![Page 44: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/44.jpg)
Summary of “how to experiment”
• Choose your variables, make them concrete
– IV, DV
• Choose your method
– Correlational or experimental
• Analyze data
– Depends on “type” of measurement
![Page 45: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/45.jpg)
Recap
• Which should you choose when?
• Correlational research
– Important, “hard to manipulate” real-life outcomes
– Could not ethically or practically manipulate
• Experimental research
– Make causal claims
– Can manipulate cleanly
Variable Outcome Internal
validity!
![Page 46: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/46.jpg)
• What does it mean to “cleanly” manipulate…?
• Hard to create conditions that only differ on one variable
– Why is it hard? What have you noticed about manipulations?
– …but its important to try…. Why?
Challenge in experimental research
Internal
validity!
![Page 47: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/47.jpg)
• If conditions differ on more than one variable
– Those other differences are potential threats to internal validity
– what things differ between rejected & walk to school other than increased need to belong?
• Pain
• Negativity
• Mood
• Anxiety
– These could be alternative explanations• If rejected were more socially sensitive, could be….
Challenge in experimental research
![Page 48: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/48.jpg)
Solutions in experimental research
• But, are ways to help ensure internal validity
• 1. Control condition similar to experimental
• The conditions are very much alike, helps to control
for some factors (alone vs with others, anxiety, etc)
– But still might differ on more than rejection: pain,
negativity, mood…
![Page 49: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/49.jpg)
Solutions in experimental research
• 2. Other additional conditions that manipulate
these confounds– Pain
– Negativity
– If they don’t increase social sensitivity
but rejection condition does…
– Mood?
– Anxiety?
![Page 50: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/50.jpg)
Solutions in experimental research
• 3. Measure alternative explanation– Pain
– Negativity
– Mood
– Anxiety
– If the rejection condition doesn’t increase these
compared to control…
– Even if it does, can control for them statistically,
and if still find a difference in social sensitivity…
![Page 51: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/51.jpg)
Solutions in experimental research
• 4. Measure variable intended to manipulate
– If experimental condition is higher on that
variable…
– Manipulation check
– Mediation analysis
0
1
2
3
4
5
6
7
Control Rejected
Be
lon
gin
g n
ee
ds
Rejection
manipulationBelonging needs Sensitivity
![Page 52: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/52.jpg)
Solutions in experimental research
• 5. Different manipulations across studies
– If always find difference…
– Multiple ways to increase NTB
• Reliving task
• Cyberball
• Living alone
• Chatroom
• More!
Convergent
validity!
![Page 53: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/53.jpg)
Solutions in experimental research
• 6. Combine with other ways of asking the
question
• Does NTB cause heighten social sensitivity?
• A) NTB scores associated with sensitivity
• B) Heighten need (starve) increases sensitivity
• C) Reduce need (feed) decreases sensitivity?
Convergent
validity!
And, if its like hunger, you could also…?
![Page 54: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/54.jpg)
Solutions in experimental research• Lucas et al.
• Priming acceptance to “satiate”– Acceptance - e.g., together went us they there
– Control - e.g., pretty was chair the skiing
• Social sensitivity
0
1
2
3
4
5
6
7
Acceptance Control
So
cia
l s
en
sit
ivit
y
![Page 55: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/55.jpg)
Questions?
![Page 56: Lecture 9: Experimental Methods](https://reader033.fdocuments.net/reader033/viewer/2022061100/629a8aeff6b1d033916f620e/html5/thumbnails/56.jpg)
SparkNotes reading for today
• Its either available
online for free
(http://www.sparkn
otes.com/psycholo
gy/psych101/resea
rchmethods/),
where you have to
click through all the
sections yourself
• or download easier
-to-read (for < $5)
• http://www.barnesa
ndnoble.com/w/res
earch-methods-in-
psychology-
sparknotes-
editors/101822024
3?ean=978141149
1663