Experimental design

Experimental design

Outlines • What is an experiment?

• Controlling the experiment

• Design categories (Experimental, Quasi-

experimental and Ex Post Facto)

• Comparison

• Meta analysis

• Examples

Experimental design

Experiment Ordered investigation that attempts to prove or disapprove a hypothesis

Questions

Hypothesis

Experiment

Analysis

Conclusion

Scientific Method Components

Experimental design

Experimental design

• So .. The hypothesized relationship isIndependent variable -> dependent variable

Experimental design

• An independent variable is one that the researcher studies as a possible cause of something else.

• Dependent variable is a variable that is potentially influenced by the independent variable.

• Manipulation: Researcher manipulates the independent variable ( treatment – intervention)

Experimental design

• So .. The hypothesized relationship isIndependent variable -> dependent variable

Experimental design

• So .. The hypothesized relationship isIndependent variable -> dependent variable Certain Drug impact on cancer

Experimental design

• A researchers can convincingly identify cause-and-effect relationships by experimental design

• Many possible factors that might cause or influence a particular condition or phenomena.

• The researcher attempts to control for all influential factors except the study factor.

Controlling for confounding variables

• 1- keep something constant• 2- Include a control group• 3- Randomly assign people to group• 4- Assess equivalence before the treatment with

one or more pretests .• 5- Expose participants to all experimental

conditions• 6- statistically control for confounding variables.

• Regarding cause-effect or causal relationships, internal Validity is the approximate truth about inferences.

• It is relevant in studies that try to establish a causal relationship.

• The key question in internal validity is whether observed changes can be attributed to your program or intervention (i.e., the cause) and not to other possible causes.

• Did the treatment cause the outcome to occur? Or were there other confounding factors that caused the outcome?

Internal Validity

• Are the findings unique to just participants we studied or could they apply to other groups?

• Refers to the extent to which the results of an experiment can be generalized across populations, time and settings.

External Validity

• Imagine that we are studying the effects of an education program in mathematics for first grade students on a measure of math performance such as a standardized math achievement test.

• In the post-only design, we would give the first graders the program and then give a math achievement posttest.

• Consider what would happen if you observe a certain level of posttest math achievement or a change or gain from pretest to posttest.

• You want to conclude that the outcome is due to your math program. How could you be wrong?

Single Group Threats

• History Threat: It's not your math program that caused the outcome, it's something else, some historical event that occurred.

• Maturation Threat: The children would have had the exact same outcome even if they had never had your special math training program. All you are doing is measuring normal maturation or growth in understanding that occurs as part of growing up.

• Testing Threat: This threat only occurs in the pre-post design. What if taking the pretest made some of the children more aware of that kind of math problem -- it "primed" them for the program so that when you began the math training they were ready for it in a way that they wouldn't have been without the pretest.

Single Group Threats

• Instrumentation Threat: Like the testing threat, this one only operates in the pretest-posttest situation. What if the change from pretest to posttest is due not to your math program but rather to a change in the test that was used?

• Mortality Threat: It means that people are "dying" with respect to your study.

• Regression Threat: Also known as a "regression artifact" or "regression to the mean" is a statistical phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated.

Single Group Threads

Design Categories

• Pre-experimental designs

• True Experimental designs

• Quasi-experimental designs

• Ex post facto designs

• Factorial designs

Design Categories

• Some are true experimental designs as such they allow us to identify cause-and-effect

• Some give alternative explanations of an observed change.

Design Categories

• All of the designs have one thing in common : clearly identify independent and dependent variable.

Design Categories

We illustrate the designs using tables that have this general format "Table"

Group 1

Group 2

Group Time ->

Pre- Experimental Design

• The cells have one of four notations:• Tx: Indicates that a treatment is presented • Obs: Indicates that observation is made • : Indicates that nothing occurs during a

particular time period. • Exp: Indicates a previous experience

Pre- Experimental Design: One-shot Experimental Case Study

The most primitive type of experiment that might be termed "research“

• The design has low internal validity • 1- the characteristics or the behavior observed after

the treatment existed before the treatment as well.• 2- influenced by other factors • 3- a single measurement or observation doesn't

guarantee that situation has change or not.

Group 1 Tx Obs

Pre- Experimental Design: One-shot Experimental Case Study

• Example .. • The design will be something like this :

Exposure to cold + Damp ground (TX)-> Child has a cold (Obs)

• One-shot experimental case study is simple to carry out, its results are meaningless.

Pre- Experimental Design: One- Group Pretest-posttest Design

• We know that a change has taken a place. But we have not ruled out other possible explanation for the change.

Group 1 Obs Tx Obs

Pre- Experimental Design: Static Group Comparison

• Involves both an experimental group and a control group.

• No attempt is made to check wither they are similar or not before the treatment so no way to know if the treatment actually causes any differences between the two groups.

Group 1 Tx Obs

Group 2 Obs

• If random assignment is used, we call the design a randomized experiment or true

experiment. • If random assignment is not used, then we have to ask a second question: Does

the design use either multiple groups or multiple waves of measurement? • If the answer is yes, we would label it a quasi-experimental design. If no, we

would call it a non-experimental design.

Types of Design

• True experimental design is regarded as the most accurate form of experimental research, in that it tries to prove or disprove a hypothesis mathematically, with statistical analysis.

• For an experiment to be classed as a true experimental

design, it must fit all of the following criteria:– The sample groups must be assigned randomly.– There must be a viable control group.– Only one variable can be manipulated and tested. The tested subjects

must be randomly assigned to either control or experimental groups.

True Experimental Design

True Experimental Design: Pretest-Posttest Control Group Design

• The pretest-posttest equivalent groups design provides for both a control group and a measure of change but also adds a pretest.

• It is important that the two groups be treated in a similar manner.

Random Assignment

Group 1 Obs Tx Obs

Group 2 Obs Obs

• The Solomon Four-Group Design is designed to deal with a potential testing thread.

• This design has four groups.• Two of the groups receive the treatment and two do not.

True Experimental Design: The Solomon Four-Group Design

Random Assignment

Group 1 Obs Tx Obs

Group 2 Obs - Obs

Group 3 - Tx Obs

Group 4 - - Obs

• Sometimes, randomness is either impossible or impractical. In those situations use quasi-experimental design.

• However, has features that can eliminate many threats to internal validity.

• Used frequently in evaluation because: – Often randomization is impossible or difficult– Ethical/legal prohibitions against randomization– No viable control group available – Inadequate resources to conduct randomization

Quasi-Experimental Designs

• To show that two groups are equivalent with respect to the dependent variable prior to the treatment, thus eliminating initial group differences as an explanation for post-treatment differences.

• Differs from experimental designs because test and control groups are not totally equivalent; equivalence on the pretest ensures equivalence only for variables that have specifically been measured.

Quasi-Experimental Designs: Nonrandomized Control Group Pretest-Posttest Design

Group 1 Obs Tx Obs

Group 2 Obs - Obs

• Threats to validity:– Partly controls for history threat(external event would

affect both groups, provided groups, provided groups are similar), maturation, instrument threats and instrumentation threats.

– However, even if groups are statistically very similar, if the intervention is given to volunteers they may behave differently than control group(due to self-selection).

– Regression threat.

Quasi-Experimental Designs: Nonrandomized Control Group Pretest-Posttest Design

• To show that, for a single group change occurs during a lengthy period only after the treatment has been administered.

• Provides a stronger alternative to “One group pretest-posttest design; external validity can be increased by repeating the experiment in different places under different condition.

Quasi-Experimental Designs: Simple time-series experiment

Group 1 Obs Obs Tx Obs Obs

• Threats to validity:– Extension of pretest-posttest design but reduces

maturation, testing, regression threats– No control group so no selection threat.– History threat is controlled partially.– if measurement changes around time of program

instrumentation threat may be present

Quasi-Experimental Designs: Simple time-series experiment

• Bolstering the internal validity of the preceding design with the addition of a control group.

• Involves conducting parallel series of observations for experiment and control groups.

Quasi-Experimental Designs: Control group, time-series design

Group 1 Obs Obs Tx Obs ObsGroup 2 Obs Obs - Obs Obs

• Showing , in a single group or individual, that a treatment consistently leads to a particular effect.

• Is an on-again, off-again design in which the experimental treatment is sometimes present, sometimes absent.

Quasi-Experimental Designs: Reversal, time-series design

Group 1 Tx Obs - Obs Tx Obs

• Showing , in a single group or individual, that different treatments have different effects.

• Involves sequentially administrating different treatments at different times and comparing their effects against the possible consequent of non-treatment.

Quasi-Experimental Designs: Alternating treatments design

Group1 Txa Obs - Obs Txb Obs

• Showing, the effect of a treatment by initiating at different times for different groups or individuals, or perhaps in different setting for a single individual.

• Involves tracking two or more groups or individuals over time, or tracking a single individual in two or more settings, for a lengthy period of time, as well as initiating the treatment at different times for different groups, individuals, or settings.

Quasi-Experimental Designs: Multiple-baseline design

Group1 - Obs Tx Obs Tx Obs

Group2 - Obs - Obs Tx Obs

• Showing , the effect of a treatment by initiating it at different times for

• Involves sequentially administrating different treatments at different times and comparing their effects against the possible consequent of non-treatment.

Quasi-Experimental Designs: Alternating treatments design

Group1 Txa Obs - Obs Txb Obs

Ex Post Facto Designs

• Sometimes we can’t manipulate some variables

– Impossible, e.g.: Characteristics

– Unethical, e.g.: Virus

• Ex post facto = after the fact

– Already happened in the past

• Possible effect of an experience/condition that

occurred in the past

• May show difference but not conclusive

Ex Post Facto: Simple Design

Group 1 Exp Obs

Group 2 - Obs

Group Time ->

• 2+ independent variables

• Simultaneously or sequential

Factorial Designs

Time ->

Factorial Design:Two-factor experimental design

Random Assignme

nt

Group 1 Tx1 Tx2 Obs

Group 2 Tx1 - Obs

Group 3 - Tx2 Obs

Group 4 - - Obs

Group Time ->

Factorial Design:Two-factor experimental design

Random Assignme

nt

Group 1 Tx1 Tx2 Obs

Group 2 Tx1 - Obs

Group 3 - Tx2 Obs

Group 4 - - Obs

Group Time ->

Factorial Design:Two-factor experimental design

Random Assignme

nt

Group 1 Tx1 Tx2 Obs

Group 2 Tx1 - Obs

Group 3 - Tx2 Obs

Group 4 - - Obs

Group Time ->

Factorial Design:Two-factor experimental design

Random Assignme

nt

Group 1 Tx1 Tx2 Obs

Group 2 Tx1 - Obs

Group 3 - Tx2 Obs

Group 4 - - Obs

Group Time ->

Factorial Design:Two-factor experimental design

Random Assignme

nt

Group 1 Tx1 Tx2 Obs

Group 2 Tx1 - Obs

Group 3 - Tx2 Obs

Group 4 - - Obs

Group Time ->

Factorial Design:Two-factor experimental design

Random Assignme

nt

Group 1 Tx1 Tx2 Obs

Group 2 Tx1 - Obs

Group 3 - Tx2 Obs

Group 4 - - Obs

Group Time ->

Factorial Design:Two-factor experimental design

Random Assignme

nt

Group 1 Tx1 Tx2 Obs

Group 2 Tx1 - Obs

Group 3 - Tx2 Obs

Group 4 - - Obs

Group Time ->

Factorial Design:Two-factor experimental design

Random Assignme

nt

Group 1 Tx1 Tx2 Obs

Group 2 Tx1 - Obs

Group 3 - Tx2 Obs

Group 4 - - Obs

Group Time ->

Factorial Design:Two-factor experimental design

• Example: Clustering using K-means

– Tx1: A different method to choose initial centroids

– Tx2: A different formula to calculate distance

Factorial Design: Combined experimental and ex post facto design

Group1 Expa

Random Assignme

nt

Group 1a Txa Obs

Group 1b Txb Obs

Group2 Expb

Random Assignme

nt

Group 2a Txa Obs

Group 2b Txb Obs

Group Time ->

Factorial Design: Combined experimental and ex post facto design

• Example: Clustering using K-means

– Expa, Expa: 2 different kinds of datasets

– Txa, Txb: 2 different formulas to calculate distance

Comparison

Meta-analyses

• Replication

• Quantitative analysis of the analyses

– Primarily statistical

Examples & Discussion