MAT 1000

Mathematics in Today's World

Last Time

1. What does a sample tell us about the

population?

2. Practical problems in sample surveys.

Last Time

Parameter: Number that describes a population

Statistic: number that describes a sample

Error: discrepancies between statistics and the parameters being estimated

Bias: in repeated samples, the sample statistic consistently misses the population parameter in the same direction

Variability: different samples from the same population may yield different values of the sample statistic

Last TimeBias and variability are possible sources of sampling error.

To reduce bias, use random sampling.

To reduce variability, use larger samples.

There are also sources of nonsampling error such as processing errors, response errors, and poorly worded questions.

Today

1. Collecting data with experiments

2. Practical issues with experiments•Confounding variables

•Randomized and comparative

•Statistical significance

Experiments

An experiment, like an observational study, is a way to collect data.

In an experiment, the people collecting the data attempt to modify, influence, or affect the individuals they are studying.

The individuals in an experiment are called subjects.

Subjects may be people, or animals, or even objects.

Experiments

The actions the experimenters take in order to affect the subjects are called treatments.

The word “treatment” means more than just medical treatments.

What do the experimenters test? Does imposing a treatment on the subjects produces a certain response or reaction.

The data produced by the experiment is evidence for or against this (cause and effect) relationship.

ExperimentsExamples of questions that could be studied with an experiment:

1. In K-12 education, will decreased class sizes lead to improved scores on a standardized test?

2. Will taking vitamin A lead to an increased lifespan?

3. Are nicotine patches an effective way to quit smoking?

ExperimentsNotice in each example we are interested in two variables.

Example 1Subjects: K-12 studentsVariable 1: Class sizeVariable 2 Standardized test scores

Experiment 2Subjects: PeopleVariable 1: Vitamin A intakeVariable 2: Lifespan

ExperimentsThe question is whether there is a cause and effect relationship between these two variables:

1. Will decreasing class size cause an increase in test scores?

2. Will increasing vitamin A intake increase lifespan?

The (hypothetical) cause is called the explanatory variable

The effect is called the response variable

Experiments

Using this terminology: experiments test the hypothesis that changes in the explanatory variable will lead to changes in the response variable. We can picture this as:

Explanatory variable

Response variable

Experiments

Will decreased class sizes lead to improved scores on a standardized test?

The explanatory variable is class size, and the response variable is standardized test scores.

Class size Test scores

Experiments

Will using a nicotine patch help people to quit smoking?

Explanatory variable: using a nicotine patchResponse variable: quitting smoking (yes/no)

Nicotine patch

Quitting smoking

Confounding

How should an experiment be run?

We could collect a group of individuals, impose a treatment and see what happens.

For example, reduce all class sizes in a school district and see if test scores go up.

Problem: if test scores increase, how do we know it was due only to the smaller class size?

If we cannot distinguish the effects of different possible causes, we say those causes are confounded.

Confounding

Here is an illustration of confounding:

Explanatory variable

Response variable

Confounding variable

Confounding

ExampleResearchers wanted to compare online college classes with traditional classes.

They compared scores on standardized tests and found that students in online classes scores just as high as students in traditional classes.

Explanatory variable: two values: traditional class or online class.

Response variable: test scores

Confounding

But, the students who enrolled in the online class tended to have more prior knowledge of the course material:

On a test given before the semester started, the average score for students in the online course was 40.70. For students in the traditional class it was 27.64.

How much of the performance of the students in the online course was due to their better preparation? We have no idea.

The variables of “taking an online course” and “being more prepared” are confounded.

Confounding

Taking an online course Test score

Being more prepared

The placebo effect

A very important example of a confounding variable is the placebo effect.

A placebo is any medical treatment which ought to have no effect on a patient’s condition.

A placebo could be a pill with no active medication in it, say a sugar pill. But a placebo could also be a wide variety of other things: creams, inhalants, injections, even faked surgeries.

Logically, if a patient is given a placebo, we expect to see nothing happen.

The placebo effect

It may be surprising to find out that patients who are given a placebo sometimes do show signs of improvement—this is called the placebo effect.

A sick person given nothing but a placebo will probably experience fewer or less intense symptoms.

It’s important to realize that this isn’t all in the patient’s imagination.

For example, it has been shown that painkillers that are administered without the patient’s knowledge are less effective—knowing you are getting a painkiller is part of what makes it work.

The placebo effect

For an experimenter testing a medical treatment, the placebo effect is a very important practical problem.

We can’t just give a bunch of smokers the nicotine patch and see what happens.

Just by giving the smokers any kind of patches (active or placebo), and telling them that the patches will help them to quit smoking, some of the smokers will probably quit.

The placebo effect

Nicotine patches

Quitting smoking

Placebo effect

Preventing confounding

What can we do to prevent confounding (from the placebo effect or any other extraneous variable)?

Our experiments should be comparative, which means they should compare one or more treatment.

Instead of giving all of the subjects nicotine patches, we give half of them a placebo (an inactive patch that looks the same).

Preventing confounding

As a result of the placebo effect, we expect that all of the subjects will show some improvement, whether or not they get an active medication or a placebo.

But now that we have two groups, we can look at whether the group getting the active treatment shows more improvement.

Having a group that gets a placebo allows us to control for the placebo effect. That’s why the group getting the placebo is sometimes called the control group.

You will also see the term placebo controlled.

Preventing confounding

But our other example of an experiment, comparing online and traditional classes, was comparative. Yet it still had confounding.

Our other issue is to make sure the treatment groups are not different in any significant way.

This is a similar problem to one we dealt with last class: how to pick a sample that represents a population.

The solution is the same: randomize.

Preventing confounding

In that earlier example, if we had randomly assigned students to an online or traditional class, then we can be confident that the best prepared students will not all be in one or the other class.

So the best design for an experiment is one that is comparative and randomized.

We choose two or more treatments, then randomly assign our subjects to one of the treatment groups. Simple random sampling is one way of making the assignments.

Preventing confounding

Medical experiments are often “double blind.” This means that neither the subjects getting a treatment, nor the doctor or nurse administering that treatment knows whether it is a placebo.

It makes sense not to tell the subject: we want to control for the placebo effect. If a patient knows they aren’t getting an active medication, there will be no placebo effect.

Why not tell the doctor?

We want all of our subjects to be treated the same. A doctor could (even unconsciously) treat subjects differently if she knows they are getting a placebo.

Statistical significance

When reading the results of an experiment, you may encounter the term statistical significance.

The results of an experiment are statistically significant if they were unlikely to have occurred by chance alone.

Think about it this way: if we take a group of students, randomly divide them into two groups, and give each group the same test, what do we expect?

The average scores for each group should be similar. But, we don’t expect them to be identical.

Statistical significance

These chance variations are always present in the outcome of an experiment. We want to see that our treatment has made a bigger difference than what we’d see by chance.

If we randomly assign half of a group of students to take a traditional class, and the other half to take an online class, we want to see a difference in the results that is greater than what we’d see just by chance alone.

If we do, then our results are statistically significant.

Statistical significance

A word of caution about the phrase “statistically significant.”

The results of an experiment could be statistically significant, but practically meaningless.

If we have a large number of subjects in an experiment, even very small differences can be statistically significant.

For example, with enough participants, we could see a very small difference which is statistically significant.

Statistical significance

So it may be that online class has a statistically significant difference in the average standardized test scores, but that the difference could be only 1 or 2 percentage points.

In that case putting all students into online classes may not be worth the expense.

So remember that “statistically significant” is something to look for in the results of an experiment, but by itself it does not mean “important.”