Discussion & Conclusions
description
Transcript of Discussion & Conclusions
1Dr. McKirnan; Foundations of
Research
Discussion &Conclusions
Methods & data
Phenomenon
Theory
Hypothesis
Results
Basic research designs.
Dr. David J. McKirnan, University of Illinois at Chicago, Psychology; [email protected]
This is a PowerPoint show.
Click through it by pressing any key.
Focus & think about each point; do not just passively click.
Note key words and phrases
2Dr. McKirnan; Foundations of
Research
Discussion &Conclusions
Methods & data
Phenomenon
Theory
Hypothesis
Results
Basic research designs.
Want to print this file for taking notes? This file is now in your downloads folder in "show" format
(ppsx). To convert it to a printing format (pptx):
Click 'esc' to leave this file Open PowerPoint Under “file” click “open”, browse to this file, click “open” In the open file click “save as” and save as pptx.
To print in notes format: Go to “file” “print’. In the dialogue box click “print what?”. Click “Handouts (3 slides per page)”
Then, come back and re-run the show
3Dr. McKirnan; Foundations of
Research
Discussion &Conclusions
Methods & data
Phenomenon
Theory
Hypothesis
Results
Basic research designs.
This module addresses:Designs without a
control group
Variations on control group designs
The overall structure of an experiment
4Dr. McKirnan; Foundations of
Research
Psychology 242, Dr. McKirnan
Week 3; Experimental designs
Basic experimental designs
This module overviews the core elements of an experimental research design.
We will discuss “pre-experimental” designs
These typically have no control group or may use existing groups
They are often used in preliminary or exploratory research
“True” experiments have several key characteristics:
A control group
Random assignment of participants to groups
Standardized or uniform procedures for each group
5Dr. McKirnan; Foundations of
Research Experimental designs and validity
We will discuss internal and external validity.
Internal validity In experiments we manipulate (induce…) the Independent Variable.
We then measure the Dependent Variable.
Experimental hypothesis: the outcome (the level of the Dependent Variable) is caused by – and only by – the Independent Variable.
Internal validity: How confident are we that the outcome was due only to the Independent Variable.
Confound: A variable other than the IV that caused or influenced the result.
Did the participants in the experimental v. control groups differ on something other than the IV?
Were the procedures biased in some way…?Confound
6Dr. McKirnan; Foundations of
ResearchExperimental designs and validity
We will discuss internal and external validity.
External validity Experimental participants are a sample of the larger population.
The experimental manipulation attempts to accurately induce the Independent Variable.
The outcome measure represents the Dependent Variable.
The experiment is conducted in a specific physical or cultural setting.
External validity:
Does the research sample accurately represent the larger population?
Do the exp. manipulation and outcome measures accurately represent the concepts underlying the Independent & Dependent Variables?
Is the experimental setting representative of how these processes work in nature?
7Dr. McKirnan; Foundations of
Research External validity: summary
The study structure & context
The research
Setting:
The Dependent
Variable
The research Sample:
Is the sample representative of the larger population?
Is this typical of the natural
settings where the phenomenon
occurs?
Does the outcome measure represent
what we are trying to explain?
Does the experimental manipulation actually create the phenomenon you
are interested in?
The Independent
Variable
8Dr. McKirnan; Foundations of
Research Overview: Basic Designs “Pre-experimental” designs: no control group
Post-Test Only Design Pre- Post- Test Design
Group assignment
Pre-test Experimental manipulation
Outcome
Experimental Observe2TreatmentObserve1
9Dr. McKirnan; Foundations of
Research Basic Designs
“Pre-experimental” designs: no control groupPost-Test Only Design Pre- Post- Test Design
Group assignment
Pre-test Experimental manipulation
Outcome
Experimental Observe2TreatmentObserve1
True (or Quasi-)experimental designs with a control group
“After only” Control group design
Pre- Post- Group Comparisons
Control Observe2ControlObserve1
10Dr. McKirnan; Foundations of
Research Basic Designs
“Pre-experimental” designs: no control groupPost-Test Only Design Pre- Post- Test Design
Group assignment
Pre-test Experimental manipulation
Outcome
True (or Quasi-)experimental designs with a control group
“After only” Control group design
Pre- Post- Group Comparisons
Multiple group comparison
Experimental Observe1 Observe2
Experimental
Control
Observe1
Observe1
Treatment 2 Observe2
Observe2
Treatment 1
Control
11Dr. McKirnan; Foundations of
Research
“Pre-experimental” designsPost-Test Only Design
Treatment MeasureGroup
Only 1 group - typically an existing group: no selection or assignment occurs.
Experimental intervention (“Treatment”) may or may not be controlled by the researcher.
Use for naturally occurring or system-wide events (e.g., group trauma, government policy change, etc.).
Measurement may or may not be controlled by the researcher.
Measure1Treatment Measure1
Group
Only one group;• only group
available?• naturally
occurring intervention?
Measurements given to all participants at baseline & follow-up
All participants get the same treatment, which may or may not be controlled by the researcher.
Pre- Post- Test Design
12Dr. McKirnan; Foundations of
Research “Pre-experimental” Designs (2)
Allow us to study naturally occurring interventions.
Advantage of “Post-” & “Pre- Post-” Designs:
e.g., test scores before and after some school change,
Crime rates after a policy change, etc.
Having both Pre- and Post measures allows us to examine change.
13Dr. McKirnan; Foundations of
Research “Pre-experimental” Designs (2)
Disadvantage of “Post-” & “Pre- Post-” Designs:
Maturation: Participants may be older / wiser by the post-test
History; Cultural or historical events may occur between pre- and post-test that change the participants
Mortality: Participants may non-randomly drop out of the study
Regression to baseline: Participants who are more extreme at baseline look less extreme over time as a statistical confound.
Reactive Measurement: Scores may change simply due to being measured twice, not the experimental manipulation.
No control group = many threats to internal validity.
14Dr. McKirnan; Foundations of
Research
Experiments“After only” Control group design
Adds a control group. Either…
Observed Groups: Naturally occurring (e.g., Class 1. v. Class 2) or Self-selected (sought therapy v. did not…).
Assigned Groups:
Randomly assign participants to experimental v. control group, or
Match participants to create equivalent groups.
Measure Dependent Variable(s) only at follow-up.
Use experimental or standard measures (e.g., grades, census data, crime reports).
Experimental
Control
Treatment 2 Observe2
Observe2Control
15Dr. McKirnan; Foundations of
Research Advantages of experimental design
“After only” Control group design
Advantage: Lessens the likelihood of confounds or threats to internal validity.
Control group Random assignment
Disadvantage: Existing or self-selected groups may have confounds.
No baseline or pre- measure available: We cannot assess change over time. We cannot assess whether the groups are
equivalent at baseline.
Experimental
Control
Treatment 2 Observe2
Observe2Control
16Dr. McKirnan; Foundations of
Research Basic Designs: True experiments (2)
Pre- Post- Group Comparisons (most common study design)
Two groups:
Observed (quasi-experiment)
orAssigned
(true experiment).
Baseline (“pre-test”) measure of study variables and possible confounds.
Group 1
Group 2
Measure 1
Measure 1
17Dr. McKirnan; Foundations of
Research Basic Designs: True experiments (2)
Pre- Post- Group Comparisons (most common study design)
The group getting the experimental condition is contrasted with a control group..
Group 1
Group 2
Measure 1
Measure 1
Treatment Measure 2
Measure2
“Post-test” follow-up of dependent variable(s);
Simple outcome Change from
baseline.
Control
18Dr. McKirnan; Foundations of
Research Basic Designs: True experiments (2)
Pre- Post- Group Comparisons (most common study design)
Group 1
Group 2
Measure 1
Measure 1
Treatment Measure 2
Measure2
Advantages: Pre-measure assesses baseline level of Dependent Variable
Allows researcher to assess change
Can find matched pairs of participants and assign each to different groups (rather than random assignment).
Can assess whether groups are equivalent at baseline.
Disadvantage: Highly susceptible to confounds if using observed or self-selected groups.
Control
19Dr. McKirnan; Foundations of
Research
More Complex Experimental Designs
Multiple group comparison
3 (or more) groups
Typically formed by Random assignment.
Multiple experimental groups, e.g. Low drug dose, High drug dose, Placebo.
or Male therapist, Female therapist, Wait list control.
Group 1
Group 2
Group 3
Measure1
Measure1
Measure1
Treatment #2
Treatment #1
Control
20Dr. McKirnan; Foundations of
Research More Complex Experimental Designs
Multiple group comparison
Measure2Group 1
Group 2
Group 3
Measure1
Measure1
Measure1
Treatment #2 Measure2
Measure2
Treatment #1
Compare: Level 1 of independent
variable from Level 2 Either / both experimental
groups from control grp.
Control
21Dr. McKirnan; Foundations of
Research More Complex Experimental Designs
Multiple group comparison
Measure2Group 1
Group 2
Group 3
Measure1
Measure1
Measure1
Treatment #2 Measure2
Measure2
Treatment #1
Advantages: Test dose or context effects:
Drug doses, amounts of psychotherapy, levels of anxiety, etc. Increasing dose effect can be tested against no dose.
Diverse conditions to test 2nd hypotheses or confounds, e.g., therapy delivered by same sex v. opposite sex therapist.
Disadvantage:
More costly and complex.
Potential ethical problem with a “no dose” (or very high dose) condition.
Control
22Dr. McKirnan; Foundations of
Research
Core components of a research study
Participant Selection
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
We will use this framework to think about the basic elements of an experiment.
Who is in our research study?
How did we recruit or sample them?
We will have at least one Experimental Group and a Control Group.
How do we assign participants to be in one or the other?
What instructions do we give?
What experimental tasks will participants be performing?
What measures might we be taking?
Experimental & control groups get different conditions.
We hypothesize that this manipulation “causes” the outcome.
What outcomes are we measuring?
What is the experiment trying to explain?
23Dr. McKirnan; Foundations of
Research Experimental design overview
Participant Selection
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
Sample
Group A Procedure Treatment Outcome
Group B Procedure Control Outcome
(Group C) (Procedure ) (Alternate Treatment?) (Outcome)
We recruit a sample of participants from the larger population.
We randomly assign them to groups to ensure the groups are equivalent at baseline.
Procedures for all groups should be exactly the same…
…except the experimental manipulation, i.e., the Independent variable.
Hypothesis: The outcome or Dependent Variable varies only by group.
24Dr. McKirnan; Foundations of
Research
Overview of true experimental designs
Participant Selection
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
Sample
Group A Procedure Treatment Outcome
Group B Procedure Control Outcome
(Group C) (Procedure ) (Alternate Treatment?) (Outcome)
Experimental group
Control group
25Dr. McKirnan; Foundations of
Research
Overview: experimental designs
Participant Recruitment
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
Sample
Group A Procedure A Treatment Outcome
Group B Procedure A Control Outcome
Group C Procedure A Alternate Treatment (?) Outcome
Does the sample well represent
the population?
External validity
• Was recruitment biased?
• Is the sample size large enough?
What form of validity is threatened by sample
bias?
What can we do to avoid that threat?
Random selection
26Dr. McKirnan; Foundations of
Research
Overview: experimental designs
Participant Recruitment
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
Sample
Group A Procedure A Treatment Outcome
Group B Procedure A Control Outcome
Group C Procedure A Alternate Treatment (?) Outcome
Does the sample well
represent the population?
External validity
Random selection
Are the groups equal at
baseline?
Internal validity
Random Assignment
• Did participants Self-select (in or out) of the study?
• Did we use existing groups?
Validity Threat?
Solution?
27Dr. McKirnan; Foundations of
Research
Overview: experimental designs
Participant Recruitment
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
Sample
Group A Procedure A Treatment Outcome
Group B Procedure A Control Outcome
Group C Procedure A Alternate Treatment (?) Outcome
Does the sample well
represent the population?
External validity
Random selection
Are the groups equal at baseline?
Internal validity
Random Assignment
Procedures the same for all groups?
Internal validity:
Lack of confounds
• Do both groups have the same expectations?
• Are participants (and researchers) really blind?
• Do we treat both groups the same?
Validity Threat?
Solution?
28Dr. McKirnan; Foundations of
Research
Overview: experimental designs
Participant Recruitment
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
Sample
Group A Procedure A Treatment Outcome
Group B Procedure A Control Outcome
Group C Procedure A Alternate Treatment (?) Outcome
Does the sample well
represent the population?
External validity
Random selection
Are the groups equal at baseline?
Internal validity
Random Assignment
Procedures the same for all groups?
Internal validity:
Lack of confounds
Independent variable faithfully
reflects the construct?
External Validity
Correct IV?
• Does the operational definition really express the construct we are interested in?
• Have we given the correct dose of the IV?
Validity Threat?
Solution?
29Dr. McKirnan; Foundations of
Research
Overview: experimental designs
Participant Recruitment
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
Sample
Group A Procedure A Treatment Outcome
Group B Procedure A Control Outcome
Group C Procedure A Alternate Treatment (?) Outcome
Does the sample well
represent the population?
External validity
Random selection
Are the groups equal at baseline?
Internal validity
Random Assignment
Procedures the same for all groups?
Internal validity:
Lack of confounds
Independent variable faithfully
reflects the construct?
External Validity
Correct IV?
Internal Validity:
Statistical testing
Groups really
different at outcome?
• Is any difference we see actually statistically significant (reliable & meaningful)?
• …or it is due to chance alone..
Validity Threat?
Solution?
30Dr. McKirnan; Foundations of
Research
Overview: experimental designs
Participant Recruitment
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
Sample
Group A Procedure A Treatment Outcome
Group B Procedure A Control Outcome
Group C Procedure A Alternate Treatment (?) Outcome
Does the sample well
represent the population?
External validity
Random selection
Are the groups equal at baseline?
Internal validity
Random Assignment
Procedures the same for all groups?
Internal validity:
Lack of confounds
Independent variable faithfully
reflects the construct?
External Validity
Correct IV?
Internal Validity:
Statistical testing
Groups really different at outcome?
31Dr. McKirnan; Foundations of
Research
Experimental design key elements Control group v. non-controlled designs Threats to internal validity:
Maturation History Mortality Regression to baseline Reactive Measurement
“Pre-experimental” designs Pre-post designs Multiple group comparisons
Overview: key terms
32Dr. McKirnan; Foundations of
Research
Overview: experimental designs
Participant Recruitment
Participant Assignment
Experimental Procedures
Experimental Treatment or Manipulation
Results
Does the sample well
represent the population?
External validity
Are the groups equal at baseline?
Internal validity
Procedures the same for all groups?
Independent variable faithfully
reflects the construct?
Groups really different at outcome?
External validity
Internal validity
Internal validity