Randomization and Controls

Homework 5

Last time we learned that when A and B are two correlated variables, there are four possible explanations:

• A causes B• B causes A• Something else, C, causes both A and B• It’s just a coincidence/ accident

Observational studies reveal correlations between two variables, and these correlations are consistent with any of our four possible explanations.

However, in the news, the correlations are often presented in causal terms, claims like “A causes B” even though that hasn’t been shown.

Eggs and Arteries

According to the article “Egg Yolks almost as bad as smoking,”

Dr. David Spence found that “people who eat egg yolks regularly have about 2/3 as much plaque buildup as smokers.”

A Causal Claim

Spence is quoted in the article saying, “Just because you are 20 doesn’t mean egg yolks aren’t going to cause any trouble down the line.”

He’s clearly suggesting that eggs cause plaque build up.

The theory outlined in the article is that eggs contain lots of cholesterol.

Alternative Explanations

But are there other explanations?

It’s very unlikely that plaque in your arteries causes you to eat more eggs. (B causes A)

However, it’s possible that there is a common cause of egg-eating and plaque. (C causes A and B)

Eggs and Bacon

Eggs are often served with unhealthy meals– omelets with bacon and cheese, cheeseburgers, fried rice, greasy Korean soup…

Maybe it’s the things we eat with eggs that cause the plaque build up.

Chocolate and Nobel Prizes

In the article, “Eat more chocolate, win more Nobels?” Dr. Franz Messerli claims to have found "a surprisingly powerful correlation” between the chocolate consumption in a country and the Nobel rate.

Chocolate and Flavanols

The theory outlined in the article is that chocolate contains flavanols; flavanols slow down age-related mental decline (though this is doubtful); and…?Well, it’s not really explained how lessened mental decline makes you more likely to win Nobels. Wouldn’t chocolate have to make you smarter and not just prevent you from being dumber?

B causes A

Dr. Messerli, according to the article, admits that “it’s possible… that chocolate isn't making people smart, but that smart people who are more likely to win Nobels are aware of chocolate's benefits and therefore more likely to consume it.”

C causes A and B

The article also quotes Sven Lidin, the chairman of the Nobel chemistry prize committee: “I don't think there is any direct cause and effect. The first thing I'd want to know is how chocolate consumption correlates to gross domestic product.” He seems to be suggesting that GDP causes higher chocolate consumption and more Nobel prizes.

The GDP Theory

Here’s what I think Lidin is suggesting:Chocolate is a luxury. Wealthy individuals are more likely to be able to afford it. Education is also a luxury. Poor people can’t afford to go to college for 10 years to get a PhD in chemistry. But you can’t win the Nobel prize in chemistry unless you’re a chemist.So he expects that the GDP or “wealth” of a country will be correlated both with chocolate eating and with Nobel prizes.

The GDP Theory

So he expects that the GDP or “wealth” of a country will be correlated both with chocolate eating and with Nobel prizes.

Wealth causes chocolate eating & Nobels.

Spurious Correlation

It’s also possible that Dr. Messerli has committed the ecological fallacy, assuming that a correlation between a country’s chocolate consumption and that country’s number of Nobel prizes means that there is a correlation between individual chocolate consumption and individual Nobel-prize winning.

Ecological Fallacy Explanation

Maybe smart people tend to avoid chocolate, because they know it can cause obesity. When they live in a country that consumes lots of chocolate they have to exercise their will power frequently. And maybe smart people + strong willpower = more Nobels. So it’s not eating chocolate but avoiding chocolate that causes Nobel prizes.

Cheating & Economic Dependence

In “Men More Likely to Cheat If They Are Economically Dependent On Their Female Partners, Study Finds” a correlation is found between a man’s economic dependence on their spouse and his likelihood of cheating on her.

Cheating & Economic Dependence

The study, authored by Christin Munsch, a sociology Ph.D. candidate at Cornell University, found:

“men who were completely dependent on their female partner's income were five times more likely to cheat than men who contributed an equal amount of money to the partnership.”

Controlling for Other Factors

Munsch does not conclude that economic dependence directly causes cheating. He finds:

“The relationship between economic dependence and infidelity disappeared when age, education level, income, religious attendance, and relationship satisfaction were taken into account.”

Root Causes

That means that if two men have• The same age• The same education level • The same income• The same level of religious attendance and• The same relationship satisfactionThen they are equally likely to cheat, regardless of their economic dependence on their spouse.

A causes C, C causes B

So the idea is that economic dependence affects one of these variables. For example:

High economic dependence↓

Low relationship satisfaction↓

Higher rates of cheating

B causes A

Are there alternative explanations?

Well, it doesn’t make a lot of sense to suppose that cheating on your partner causes high economic dependence on your partner.

After all, the study found that women who are highly economically dependent are less likely to cheat.

C causes A and B

Still, there are other possible explanations of the data.

It might be that old people are more traditional. Older men are less likely to be dependent on their female partners, because this is not traditionally acceptable. In addition, older men are less likely to cheat, because they have traditional values about marital fidelity.

The Age Theory

Younger people are less traditional. Young men are more likely to be economically dependent on women and more likely to cheat:

Economic Dependence↑

Age↓

Cheating

Quitting Smoking & Beating Cancer

According to the article, “With lung cancer, quitters do better than smokers”:“Younger people with advanced lung cancer who quit smoking more than a year before their diagnosis survive longer than those who continue smoking, according to a new study.”

Quitting Smoking & Beating Cancer

The study finds: “Among smokers with stage 1 or 2 lung cancer, for instance, 72% survived at least two years, compared to 93% of the never-smokers and 76% of people who'd kicked the habit a year or more before diagnosis.

Only 15% of smokers with stage 4 disease survived two years, while 40% of never-smokers and 20% of former smokers did.”

Smoking Makes Cancer Worse?

One interpretation is that smoking while you have cancer makes the cancer worse, or makes tumors grow faster. People who quit don’t have this effect.

But the study also found: “After adjusting the numbers for factors such as age, race and radiation treatment, the researchers determined that quitters were just as likely to die from the early-stage cancers as were current smokers.”

Other Factors

This means that two people, at the same stage of lung cancer, who also had:• The same age• The same race• The same amount of radiation treatmentWere equally likely to die of cancer, even if one had quit smoking and the other had not.

A Morbid Possibility

Here’s a possible common cause: a desire for death. People who wish they were dead are more likely to keep smoking and less likely to undergo radiation therapy when they find out they have cancer.

People who want to live are more likely to quit, and more likely to undergo radiation treatment.

The Death-Wish Theory

Smoking↑

Desire for Death↓

No Radiation Therapy

Overall Lessons

Observational studies only reveal correlations, not causation.

These studies are often reported in causal terms, which is misleading.

There are often alternative explanations for the data, but we can’t know whether they are true either.

CONTROLLING AND RANDOMIZATION

Controlled Experiments

Suppose I believe that eating chocolate makes you smarter.

Maybe I have some evidence, in the form of observational studies that show a correlation between chocolate consumption in a country and the number of Nobel prizes won by that country.

But there are alternative theories:

• Smartness causes chocolate eating• Wealth causes smartness and chocolate eating• Chocolate avoiding causes smartness• Etc.

Experimental Design

I can rule out these possibilities with a well-designed experiment.

What I want is two groups: one group (the experimental group) that eats chocolate because I tell them to, and another group (the control group) that does not eat chocolate, because I tell them not to.

Not: B causes A

If the experimental group improves in intelligence over the course of the experiment, I know that this is not because higher intelligence leads to more chocolate consumption (even if that is true).

In my experiment, intelligence does not cause chocolate consumption, I do. I am the experimenter and I say who eats chocolate.

Controlling for

Additionally, if I make sure to put equal numbers of rich people in both groups, and equal numbers of middle-class people, and equal numbers of poor people, then I can make sure that improvements in the experimental group are not due to wealth: both groups have the same distribution of wealthy and non-wealthy people. This is called controlling for wealth.

Randomization

Ideally, an experiment controls for as many variables as possible.

To a large extent, this is done by randomly assigning individuals in the study to either the control group or the experimental group. This way, the members of the group are less likely to share features other than chocolate eating.

Randomization

Randomization is not the only tool for controlling for confounding variables, and for certain variables, it can’t help.

For example, suppose I want to test whether seeing pictures of babies makes people happier.

Babies and Happiness

I randomly assign participants in the study to the control group and the experimental group.The control group takes a happiness questionnaire.The experimental group looks at baby pictures and then takes a happiness questionnaire.

A Crucial Difference

Suppose that the experimental group rates higher on the happiness questionnaire. Does this mean that baby pictures cause happiness?

No. The control group didn’t get to look at any pictures. Maybe they got bored, and boredom makes you less happy. We should give the controls pictures other than babies.

Maximal Similarity

In general, the control condition and the experimental condition should be as similar as possible, and differ only in the variable being tested.For example, if the experimental group is given the happiness test by a beautiful woman and the control group is tested by a grumpy professor, that might be the real reason for a difference in scores, not the baby pictures.

It can be difficult or impossible to make the control and experimental conditions similar.

If you want to study the effects of exercise, how do you make exercise and non-exercise similar?

BLINDING AND DOUBLE-BLINDING

“Blinds”

In experimental studies, we say that the participants are blind if they do not know which group they are in: the control group or the experimental group.

Again, it’s not always possible to have blind participants, but this is considered best practice.

The Placebo Effect

Why is blinding important? For several reasons:

First, the mind has a pretty powerful effect on the body. People who think they’re receiving an effective treatment (even if they aren’t) are more likely to get better People who think they’re not receiving an effective treatment (even if they are) are less likely to get better.

Placebo Controls

So we don’t test a drug by giving it to the experimental group and giving nothing to the control group. Then the control group knows it’s the control group, because it gets no pills!

Instead, we give the control group fake pills with only sugar in them, and we don’t tell them that the pills are fake.

Subject Bias

A second reason that blinding participants is good is that people who believe they are getting an effective treatment are more likely to say they’ve gotten better (even if they haven’t) and people who believe they are not getting an effective treatment are more likely to say that they haven’t gotten better (even if they have).

Subject Bias

So, for example, if you give a group of people a fake pain medication and ask them whether it helps their pain, they might reason:“Well, I’m supposed to feel better, so I probably did get a little bit better.”

Deception and Blinding

One common way of making sure subjects don’t know which condition they’re in is by lying to them about what you’re studying.

You might tell people that you’re studying math ability, when what you’re really doing is studying the affects of cold rooms on math ability.

Improper Blinding

One study of medical experiments found that studies with “improper” blinding procedures (where the subjects could find out which group they were in) exaggerated effects by 17%.

Blind Experimenters

Ideally, in experiments the researchers are blind to which group subjects are in.

This prevents the experimenter from accidentally indicating to the subjects which group they are in.

It also prevents experimenter bias.

Clever Hans

Clever Hans was a horse that was supposed to have amazing mathematical abilities. It was claimed that he could add, subtract, multiply, divide, read, spell, and understand German.

Clever Hans

Hans’ trainer would ask him questions like:

“If the eighth day of the month comes on a Tuesday, what is the date of the following Friday?”

Then Hans would tap out the answer with his hoof.

The Clever Hans Effect

The psychologist Oskar Pfungst investigated Hans and discovered that Hans could only solve problems when (a) The questioner knew

what the answer was and

(b) The horse could see the questioner.

The Clever Hans Effect

Hans got math questions right 68% of the time when the questioner knew the answer but only 6% of the time when the questioner didn’t.

Hans was able to tell what people wanted him to do, and he would keep stamping his foot if it looked like people wanted him to go on, and he would stop when it looked like they were happy and about to reward him.

Conforming to Expectations

That’s why double-blinding is important:

If subjects know what the experimenter expects of them, their actions will conform to those expectations.

Experimenter Bias

If experimenters want a certain outcome, they can record or interpret the experimental results in a biased way.

For example, if I’m studying a blood pressure medication, and I know a subject is in the experimental group when I take his blood pressure, this might bias the results.

Experimenter Bias

If the reading is high, I might take his blood pressure a second time “just to make sure.”If the second reading is low, I have biased the results. Would I have taken a second reading for someone in the control group?

Randomization

An experimenter can also bias an experiment when it is not appropriately randomized.

If the experimenter decides who goes in the control group and who goes in the experimental group, she can (consciously or unconsciously) put people who are more likely to get better anyway in the experimental group.

Blind Randomization

But it is not enough to simply randomize, the experimenter must be blind to the randomization process itself.

For example, one method of randomization is to assign the first person who signs up to the control group, the second to the experimental group, the third to the control…

Blind Randomization

Now suppose someone really sick shows up on an “odd” time and should be assigned to the experimental group.

The experimenter might convince the person that the experiment wasn’t appropriate for them, and thus make it more likely that only healthy people were in the experimental group.

This Matters

Studies have shown that flawed randomization methods overstate effects by 30%!

That’s a huge effect, and in the case of drug trials, overexaggerated effects can cost lives.

SUMMARY

Benefits of Controlled Experiments

Today we learned that a well-designed controlled experiment can:

• Control for confounding variables• Rule out common cause and reverse cause

explanations

Elements of Good Design

Well-designed experiments should:

• Assign subjects to control/ experimental groups randomly

• Not let subjects know which group they’re in• Not let the experimenter know which group

subjects are in