Post on 02-May-2018
Experiments & Observational Studies
Chapter 13
We’ve developped a new rabbit food, HIPPITY HOP.
Hippity Hoprabbit food
Makes fur soft and shiny!
Increases energy!
100% of daily vitamins &
essential oils!
Experimental unit -
The single individual (person, animal, plant, etc.) to which the different treatments are assigned
Factor -
The explanatory variable(what you’re changing for each subject)
Level -
A specific value for the factor(each specific option for a factor)
For our Hippity Hop example:
- the rabbits
- the type of rabbit food we give to each rabbit
- probably 2 levels: Hippity Hop, and some other type of rabbit food that we’re going to compare Hippity Hop to in this experiment
Treatment -
A specific experimental condition applied to the unitsmade up of factors
at different levels
Response Variable -
what you are measuring at the end of an experiment(after the treatments have been applied)
You must be specific!!!!
- In the rabbit food example, we have 2 treatments
- hair softness and shine?amount of energy?
Example 1: A farm-product manufacturer wants to determine if the yield of a crop is different when the soil is treated with three different types of fertilizers.Fifteen similar plots of land are planted with the same type of seed but are fertilized differently. At the end of the growing season, the mean yield from the sample plots is compared.
Experimental Units?
Factors?
Levels?
How many treatments?
Response variable?
plots of land
type of fertilizer
fertilizer types A, B, C
3
yield of crop
Example 2: A consumer group wants to test cake pans to see which works the best (bakes evenly). It will test
aluminum, glass, and silicone pans in both gas and electric ovens.
Experimental Units?
Factors?
Levels?
How many treatments?
Response variable?
cake batter
Two factors - Type of pan & Type of oven
Type of pan has 3 levels (aluminum, glass, silicone), and type of oven has
2 levels (electric, gas)
6
how evenly the cake bakes
Experimental DesignIn 2007, deaths of a large number of pet dogs and cats were ultimately traced to contamination of some brands of pet food. The manufacturer has checked and tested the food, and now claims the food is safe, but before it can be released to the general public, an experiment to test whether the food is now truly safe for dogs and cats to eat must be conducted.
What would the treatments be?
New food from the company, and another that we are certain is safe
What would the response variable be?
Health of the pets, as assessed by a veterinarian
Experimental Design
A group of 32 dog owners have volunteered their pets for this experimental study. Explain how you would carry out a completely* randomized experiment to see if the new food is safe for dogs to eat. The dogs will eat the assigned food for a period of 6 weeks.
*completely randomized experiment means NO BLOCKING ALLOWED!! I’ll talk about
what this means later
When explaining an experimental design, you must: 1. Describe how you will randomly assign your subjects to
the different treatment groups.2. Say what each of the treatment groups will do.3. Explain what response variable you will
evaluate/compare at the end of the experiment.
Remember to … BE SPECIFIC!!!
COMPLETELY RANDOMIZED DESIGN
Group of 32 dog
volunteers
Group 1: 16 dogs
Group 2: 16 dogs
Treatment 1: dogs eat new
food for 6 weeks
Treatment 2: dogs eat “safe”
food for 6 weeks
Compare health of dogs, to be
evaluated by a veterinarian
Completely randomized experiment means NO BLOCKING ALLOWED!
Randomassignment
(explaining the randomization procedure)● Number the dogs with a unique number from 1 to 32.● Use a RNG on a calculator/computer to pick 16 UNIQUE random numbers
from 1 to 32 (ignore any repeats).● The dogs whose numbers are chosen will be placed in treatment group 1.
The remaining dogs will be placed in treatment group 2.
OR
● Using a RNG on calculator/computer, RANDOMLY assign each dog a unique number from 1 to 32.
● The dogs with numbers 1-16 are placed in treatment group 1. The rest of the dogs will be placed in treatment group 2.
You MUST explain the randomness procedure - even if you use a diagram to design/describe the experiment!
You may also describe the experiment all in paragraph form, which may look something like this...Give each of the 32 dogs a unique number from 1 to 32. Then use a random number generator to select 16 unique, random numbers from that range.
The 16 dogs whose numbers are selected will be assigned to Treatment Group 1, and will eat the new dog food for 6 weeks. The remaining 16 dogs will be assigned to Treatment Group 2, and will eat the “safe” dog food for 6 weeks.
After the 6 weeks are over, we will compare the health of the dogs, as assessed by a veterinarian.
Why is randomization
important?
In theory, randomization...
● ‘equalizes’ all other variables● gives us treatment groups
that are more or less the same
-BUT-
● Variation WILL occur!!Sometimes we end up with ‘unfair’ groups. :(
This is why we replicate the experiment (perform the experiment a number of times, with different groups of subjects, in different
locations, etc).
Principles of Experimental Design1. Control
Make conditions as similar as possible for all treatment groups (aside from the actual treatments, of course).
If we observe a difference between groups, we want to be able to say that difference is because of the treatments, and not something else!
2. Randomization
The use of chance to assign subjects/units to treatments
3. Replication
Repeat the experiment on many subjects/in different locations/etc.
Replication is also helpful because, in most experiments, we’re dealing with a group of volunteers, NOT a random sample.
Replication helps us apply our results to a larger population.
BLOCKING! (A form of control)
Of the 32 dogs, 12 are poodles, and 20 are German Shepherds.Explain the changes you would make to your previous design by incorporating blocking.
BLOCKING! (A form of control)
Of the 32 dogs, 12 are poodles, and 20 are German Shepherds.Explain the changes you would make to your previous design by incorporating blocking.
“We will block by breed of dog (separate poodles from German Shepherds, before we start assignment of treatments), because ….
BE SURE TO EXPLAIN WHY!!(think back to our reasoning for taking a stratified random sample - it’s similar reasoning here!)
Group of 32 dog
volunteers
12 poodles
20 German Shepherds
Block by breed!
Group 1: 6 poodles
Group 2: 6 poodles
Group 3: 10 German Shepherds
Group 4: 10 German Shepherds
Random!
Random!
Treatment 1: dogs eat new food for 6 wks
Treatment 2: dogs eat ‘safe’ food for 6 wks
Treatment 1: dogs eat new food for 6 wks
Treatment 2: dogs eat ‘safe’ food for 6 wks
Compare health of poodles
Compare health of German
Shepherds
YIKES!Plus, remember we also need to explain our randomness procedure for assigning subjects to treatment groups!
Blocking is to experiments as__________________ is to sampling.stratifying
Is the difference I observed in my results…“statistically significant?”
What does that even mean?!?
statisticallysignificant
When an observed effect is
SO LARGEthat it would rarely occur by chance
we call that effect ‘statistically significant.’
SAT PREP CLASSES!● Higher SAT scores in ONLY 6 weeks!● top-secret far-eastern study strategies!● 3 sessions a week, only 5 hours per session!● All the other kids are being forced into it by their
parents, so why not?!● COST: only $30,000!! What a bargain!● Scores will improve, GUARANTEED, or your
money back!
SAT PREP CLASSES!
Group of 20
students
take this class
mean score improvement of …
18 points!!!It worked!!!
CONTROL GROUP
a group that is used to compare the factor against
Can be a placebo, or the “old”/current item
SAT PREP CLASSES!
Group of 20
students
Group A: These kids care about their
grades, so they choose to take this SAT prep class Group A scored
WAYYYYYYYYhigher!
IT WORKED!!Group B:
These kids DON’T CARE about their grades, so they DON’T
take the prep class
ConfoundingTwo variables are CONFOUNDED when they BOTH have an association with the response variable.
(And we cannot tell which of the two variables is actually responsible for the effect.
Or maybe it’s both? We can’t tell!!)
Same idea as confounding …
But a lurking variable is something that we, the researchers, did not
consider before we began the experiment.
And it ended up having an important effect on the response.
(Usually this means … whoops, let’s start over and do some blocking…)
Lurking Variable
ConfoundingTwo variables are CONFOUNDED when they BOTH have an association with the response variable.
(And we cannot tell which of the two variables is actually responsible for the effect.
Or maybe it’s both? We can’t tell!!)
Same idea as confounding …
But a lurking variable is something that we, the researchers, did not
consider before we began the experiment.
And it ended up having an important effect on the response.
(Usually this means … whoops, let’s start over and do some blocking…)
Lurking Variable(an example of possible confounding)A professor at Cornell wants to know the effect of his teaching style, so he hands
out student evaluations for his students to fill out at the end of the semester.
Fall and Spring terms - a total of 472 students
His teaching was identical, exceptsubdued in the fall… enthusiastic in the spring!
Average response?Fall - 2.93
Spring - 4.05
Could the weather have been confounding?
According to Newsweek“Of all pre-college curricula, the highest level of mathematics one studies in secondary school has the strongest continuing influence on bachelor’s degree completion. Finishing a course beyond the level of Algebra 2 (for example, trigonometry or pre-calculus*) more than doubles the odds that a student who enters postsecondary education will complete a bachelor’s Degree.”
Propose a study design that might enable us to draw this conclusion. Why would it be difficult (and probably unethical) to perform this study as an experiment? Can we randomly assign students to take certain math classes, or to NOT
take certain math classes? Are there confounding variables that affect a student’s success in college,
aside from what math classes they take in high school?
about control...
testing a new headache medicine...
Group of 40
volunteers
Group 1: 20 patientsTake the pill we’re
testingCompare
number ofheadaches Group 2: 20 patients
NO PILL Sigh… Look at me, I didn’t get “the good
stuff.” I’m such a loser, I think I’ll go get a
headache now.
PLACEBOa “dummy” treatment that has no effect on the response variable
(well … we say ‘no effect’ but there is such a thing as the ‘placebo effect’)
Basically, sometimes people actually have
physical results simply because they’re
getting SOME kind of treatment. Even if it’s
not the real thing, people THINK it is … and that’s somehow
enough?
testing a new headache medicine...
Group of 40
volunteers
Group 1: 20 patientsTake the pill we’re
testingCompare
number ofheadaches Group 2: 20 patients
Take a placebo (fake pill)
BLINDINGsubjects do not know which treatment they are getting
DOUBLE-BLINDneither the subjects, nor the evaluator* knows which treatment each subject received
*the evaluator - means whoever checks the subject for improvement, but this also includes the person who administers the treatment, or cares for the subject during the experiment - basically, anybody who has direct contact with the subject
We can accomplish blind/double-blind experiments by making sure that our treatments
look, smell, feel, taste exactly the same (or as much as possible).
A pharmaceutical company is testing a new headache medicine pill, and wishes to test it in an experimental study against a control group.
Is it possible for the experiment to be blinded? double blinded?
Sure! Just make sure the real pill and the placebo pill both look, taste, feel, etc. exactly the same!
A cutting-edge treatment for combatting brain tumors involves drilling tiny holes in the patient’s skull and injecting a drug cocktail into the brain.
Is it possible for the experiment to be blinded? double blinded?
Is it ethical?Uh, yeah, we’ll just drill tiny holes in the control group patients’ brains
and inject … water? I’m sure everybody will be cool with that.
LAST EXAMPLE!(I promise)
Example of a study on stomach ulcers...In 1958, an initial experiment of a very creative treatment for stomach ulcers involved:
● Anesthetizing the patient● Putting a balloon in the patient’s stomach● Filling the balloon with freezing coolant
In this initial experiment, there were 24 patients. All were cured.
This treatment became very popular, even more so than surgery, but there
were some SKEPTICS …. Could this be the result of placebo effect?
New experiment, 1963A controlled experiment was performed.
A group of 160 ulcer patients were divided into 2 groups -
82 received the gastric freezing treatment - 34% improved
78 received a placebo treatment - 38% improved
So … Gastric freezing was NO better
than a placebo!
We NEED comparison in experiments - otherwise it’s really hard to tell when a treatment is truly working, or when other factors are causing the change.