Lies, damned lies, and surveys

Lies, damned lies ... and surveys

Beatrice MotamediThe Urban School of San FranciscoSpring JEA/NSPA National High School Journalism ConventionApril 2014

Wednesday, April 9, 14

“Lies, damned lies, and statistics” is a phrase popularized by Mark Twain, the American humorist, who wrote in “Chapters from My Autobiography” that "(f)igures often beguile me ... particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: 'There are three kinds of lies: lies, damned lies, and statistics.'"

What’s the last survey you took for your newspaper or website?

How many people responded?

What did you learn?

What would you like to do better?

Let’s BYOS (build your own survey)

• Find a person whose eye color is the same as yours and has the same kind of smartphone as you do (iPhone or Android)

• Brainstorm a question you’d like to ask right here in San Diego. How many students surf? Who believes in global warming? Should texting while driving be a crime? Choose a topic that actually interests YOU.

• Take out your smartphone and download an app: Instasurvey (Android) or Surveymonkey (iPhone)

• Set up a free account and go out with your partner/ask your question. Try to capture at least one answer using your app. Be back in 5 minutes.

let’s whip

what did you ask?

how did it go?

Glossary

• Poll: you ask one question, usually yes/no or multiple choice (what you just did)

• Survey: you ask several questions with various question types (multiple-choice, open/closed, short-answer essay, etc.).

• Random sample: a selection from a population, based on chance, and with an equal probability of being selected

• Respondents: those who actually respond to the survey (not the same as those you sample)

• Time/manner: when and how a survey is conducted, e.g., a Surveymonkey from April 7-10

• Response bias: a preference that results from problems in the surveyprocess, e.g., using leading questions, or not allowing answers to sensitive q’s to be confidential

• Sampling error: the variation in data among samples

How not to do a survey*

Click here for “RNC Young

Voters Survey,” Stephen Colbert,

4/4/13

*with apologies to young Republicans in the audience (young Dems probably have a survey just as bad as this one)

This video captures many typical survey mistakes, including non-response bias (you can’t find the survey), response bias (poorly written or leading questions) and no control over the sample (Colbert is obviously over 30 years of age but takes the survey anyway).

What’s wrong here?

Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.

too many q’s

too many assumptions (bias)

too many q’s

closed q, not open

too many q’s

closed q, not open

you woke this guy up ... for this?

Our goal: A baby survey you can use right here

• Population sample — WHO to ask

• One closed question — WHAT to ask

• One open question — ditto

• A platform — HOW you will ask your questions

Our goal for this presentation: You will leave with the beginnings of a survey, including a topic, two types of questions, a plan on how you will collect your sample, and a web-based platform that makes it all easier.

A nation of question-askers

Examples of some of the long-running established ways in which Americans ask each other questions. George Gallup began systematically surveying Americans almost a hundred years ago.

My school of question-askers

February 2013 Urban Legend

Examples from my school, including a student cartoon that expresses frustration and fatigue over constant question-asking.

A new industry: Web-based surveys and polls

Presidents are born ...

... and live by polls

Poll data at nyt.com (left) and realpolitics.com (above) as accessed on on 12/18/13.

Just one day ... at Real Clear Politics.com

Selections from the Pew “Fact Tank”

Pew regularly surveys teens on technology, religion, social trends, sexuality, and more.

Making surveys sexy

Nate Silver used his column to predict results in the 2008 and 2012 presidential elections with astonishing accuracy, correctly predicting the winner in all 50 states in 2012. Analyzing survey data is his forte. Now at ESPN, Silver says he practices “data journalism.”

Why surveys are powerful — and scary

“When they verify the popularity of an idea or proposal, surveys and polls provide persuasive

appeals because, in a democracy, majority opinion offers a compelling warrant:

A government should do what most people want.”

Source: “Everything’s An Argument” (2007: St. Martin’s Press)

“It always makes sense, though, to push back ... especially (when a poll) supports

your own point of view.”

Questions to ask

• Have you surveyed enough people to be accurate?

• Are those people representative of the selected population as a whole?

• Did you choose them randomly — not selecting those likely to say what you want to hear?

• Does the wording of your questions intentionally or unintentionally create bias or skew results?

• Have you described the results accurately and fairly? Does your story stick to what you asked, and only what you asked?

Here are questions you should ask yourself, whether you’re doing or reading a survey. Questions adapted from “Arguments Based on Facts and Reason” in “Everything’s An Argument” (2007: St. Martin’s Press).

Three tips

Three tips (and three deadly sins)

Three tips (and three deadly sins)• Get a random sample. It

should be representative also — not just people who will answer the way you want them to.

undercoverage

• Avoid bias. Ask a variety of questions (open/closed, multiple choice, short answer). Question order matters; begin with basics. Avoid leading questions.

undercoverage

response bias

• Maximize response rate. Write good questions, not too long. Use a platform that makes it easy. Analytics help.

undercoverage

response bias

• Maximize response rate. Write good questions, not too long. Use a platform that makes it easy. Analytics help.

undercoverage

non-response bias

response bias

#1: Get a random sample

This analysis by Jeanne Curran and Susan R. Takata (February 2002) shows how Americans — and the media — relied on telephone polls that failed to pick up Truman’s strength among voters who were less likely to be reached that way (in 1948, a majority of Americans). The resulting Chicago Tribune headline is a journalism classic, but for all the wrong reasons.

Surveymonkey Blog tips

Source: Surveymonkey Blog

Avoiding non-response bias means getting people to answer your survey. First you have to figure out how many people that needs to be. The more people you survey, the lower your margin of error.

How much error will you tolerate? How would you find that many people?

In order to get a solid result here at the convention, how many students would you have to survey more than one in two students. Can you do that? How? What strategies might you use?

Strategies

• Get a “concrete sampling frame” — a list of all of the students here (or at your school). Use Excel to randomize/select your sample, e.g., every other student.

• Design a random reporting strategy: Get five reporters to stand in five places here (or at your school). Each reporter spends one hour surveying every other person who walks by.

• Use advisory: Ask teachers to pass out surveys so that all grade groups are covered (can’t use this one very often)

• Be social: Embed your survey on Facebook, or use your smartphone and go to the dance (and ask every other person who shows up to take your survey)

Potential problems are double-counting, failure to randomize for age/gender, non-response bias (some people don’t answer email surveys; some won’t answer surveys unless they’re emailed), etc.

#2: Avoid bias

Voluntary response bias happens when respondents self-select, e.g., viewers of American Idol text their votes to American Idol. In season 3, Fantasia Barrino beat Diana DeGarmo by 1.3 million out of a record 65 million votes cast. But it was third-place winner Jennifer Hudson who really won. A student journalist might get similarly bad results by surveying athletes only about school athletics. You have to cast a wider net.

#2: Avoid bias

Question types: Choose a variety

closedopen

also: filter questions, e.g., have you ever smoked

marijuana (if not, then you can’t take the marijuana

smokers’ survey)

Same question, three types

multiple: good for a poll/quick react, especially if

localizing

matrix: more depth/better for a feature on tech and teens vs. other groups

localizing

text: could go either way (thoughtful, or tossed off).

If you get names, you could follow up.

Watch your words

Question #1 emphasizes cost (“at public expense”) while Question #2 emphasizes choice (“any school, public or private”). Changing the emphasis reverses the results — if you emphasize cost, most will say no, but if you emphasize choice, most will say yes. Source: Friedman Foundation School Choice poll, 2005.

Watch your words

“Obamacare” or “Affordable Care Act”?

Source: “Polling Matters,” Gallup.com, 11/20/13

brainstorm two questions, of different types

#3. Maximize response rate

analytics here

There’s always a tradeoff between anonymity and analytics. Anonymity can boost your response rate (sometimes higher esp. with controversial topics) but analytics can suffer (you can’t nag someone you can’t find). The partial surveys above on online censorship will stay that way because I can’t respond to them. Rats!

Analyzing results

• Be humble. You have not surveyed the universe, just one group of people, and your questions could have include unintentional bias.

• Write the breadbox: When and where the survey was conducted, how many in the sample, how many responded, margin of error (if you know).

• Never make a survey your one-and-only piece of evidence. Support/explain with quotes, anecdotes, observation also.

• Compare/contrast your results to other surveys — five years ago, by another group, in another state, by your same paper, etc. Put the data in perspective.

Putting results into words — rhetorical strategies

• Set up/attribute: “According to a DATE survey of HOW MANY (people, students, marijuana smokers) conducted WHEN, xx PERCENT of the HOW MANY respondents (answered, replied, responded) that ...” (and now tell us what the survey says).

• Use the right verbs — surveys show, indicate, hint, suggest, point towards, reveal, reflect, appear to or contrast with ... but they rarely prove.

• Repeat the exact language you used in your questions, because that is what the respondent answered (not a paraphrase).

• State the biggest finding first, then dig into the lesser ones. Don’t rush.

Beware of false comparisons

“In ‘The Marriage Crunch,’ (1986, Newsweek) reported on new demographic research from Harvard and Yale predicting that white, college-educated women who failed to marry in their 20s faced abysmal odds of ever tying the knot. According to the research, a woman who remained single at 30 had only a 20 percent chance of ever marrying. By 35, the probability dropped to 5 percent. In the story's most infamous line, NEWSWEEK reported that a 40-year-old single woman was "more likely to be killed by a terrorist" than to ever marry. That comparison wasn't in the study, and even in those pre-9/11 days, it struck many people as offensive. Nonetheless, it quickly became entrenched in pop culture.” —Daniel McGinn, June 2006, Salon.com

A 40-year-old single woman “more likely to be killed by a terrorist” than to marry?

Lies, damned lies, and surveys

Education

Transcript of Lies, damned lies, and surveys

Lies, Damned Lies, and Statistics - · PDF fileThe title of this course “Lies, Damned Lies, and Statistics’ is part of a famous ... may not show us the whole picture. ... illustrates

« Lies, damned lies and criminal statistics ...

DevOps Metrics - Lies, Damned Lies and Statistics

Damned Lies and Statistics

Lies, Damned Lies and Statistics

Lies, Damned Lies and Statistics: How to Make Your Data ...

Lies, Damned Lies, and Statistics

" There are three kinds of lies : lies, damned lies, and statistics "

Lies, damned lies & statistics Communication Research week 10.

Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]

Iwsm2014 lies damned lies & software metrics (charles symons)

Lies, Damned Lies, and Pretty Pictures

Lies, Damned Lies and the 2015 Intergenerational …...Lies, Damned Lies and the 2015 IGR (i) Abstract The Australian Government releases an ‘intergenerational report’ at least

Lies, Damned Lies, and Substrings

Lies, Damned Lies, and Primary Sources: The Lost History of Prester John

DAMNED LIES AND STATISTICS - The Eye Lies... · 2020. 1. 17. · Damned Lies and Statistics offers a useful guide for engaging withtheirtroublesomeworld.Despitethetemptationtobecyn-ical,

Lies, damned lies, and RCT : une expérience J-PAL sur le ...

Lies, Damned Lies & Internships: Introduction

Lies, Damned Lies and Software Analytics: Why Big Data Needs ...

Lies, Damned Lies + Email Design : Seattle Interactive Conference 2013