Lies, damned lies, and surveys
date post
13-Jul-2015Category
Education
view
86download
1
Embed Size (px)
Transcript of Lies, damned lies, and surveys
Lies, damned lies ... and surveys
Beatrice MotamediThe Urban School of San FranciscoSpring JEA/NSPA National High School Journalism ConventionApril 2014
Wednesday, April 9, 14
Wednesday, April 9, 14
Lies, damned lies, and statistics is a phrase popularized by Mark Twain, the American humorist, who wrote in Chapters from My Autobiography that "(f)igures often beguile me ... particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: 'There are three kinds of lies: lies, damned lies, and statistics.'"
Whats the last survey you took for your newspaper or website?
How many people responded?
What did you learn?
What would you like to do better?
Wednesday, April 9, 14
Lets BYOS (build your own survey)
Find a person whose eye color is the same as yours and has the same kind of smartphone as you do (iPhone or Android)
Brainstorm a question youd like to ask right here in San Diego. How many students surf? Who believes in global warming? Should texting while driving be a crime? Choose a topic that actually interests YOU.
Take out your smartphone and download an app: Instasurvey (Android) or Surveymonkey (iPhone)
Set up a free account and go out with your partner/ask your question. Try to capture at least one answer using your app. Be back in 5 minutes.
Wednesday, April 9, 14
lets whip
what did you ask?
how did it go?
Wednesday, April 9, 14
Glossary
Poll: you ask one question, usually yes/no or multiple choice (what you just did)
Survey: you ask several questions with various question types (multiple-choice, open/closed, short-answer essay, etc.).
Random sample: a selection from a population, based on chance, and with an equal probability of being selected
Respondents: those who actually respond to the survey (not the same as those you sample)
Time/manner: when and how a survey is conducted, e.g., a Surveymonkey from April 7-10
Response bias: a preference that results from problems in the surveyprocess, e.g., using leading questions, or not allowing answers to sensitive qs to be confidential
Sampling error: the variation in data among samples
Wednesday, April 9, 14
How not to do a survey*
Click here for RNC Young
Voters Survey, Stephen Colbert,
4/4/13
*with apologies to young Republicans in the audience (young Dems probably have a survey just as bad as this one)
Wednesday, April 9, 14
This video captures many typical survey mistakes, including non-response bias (you cant find the survey), response bias (poorly written or leading questions) and no control over the sample (Colbert is obviously over 30 years of age but takes the survey anyway).
Whats wrong here?
Wednesday, April 9, 14
Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.
Whats wrong here?
too many qs
Wednesday, April 9, 14
Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.
Whats wrong here?
too many qs
too many assumptions (bias)
Wednesday, April 9, 14
Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.
Whats wrong here?
too many qs
too many assumptions (bias)
closed q, not open
Wednesday, April 9, 14
Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.
Whats wrong here?
too many qs
too many assumptions (bias)
closed q, not open
you woke this guy up ... for this?
Wednesday, April 9, 14
Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.
Our goal: A baby survey you can use right here
Population sample WHO to ask
One closed question WHAT to ask
One open question ditto
A platform HOW you will ask your questions
Wednesday, April 9, 14
Our goal for this presentation: You will leave with the beginnings of a survey, including a topic, two types of questions, a plan on how you will collect your sample, and a web-based platform that makes it all easier.
A nation of question-askers
Wednesday, April 9, 14
Examples of some of the long-running established ways in which Americans ask each other questions. George Gallup began systematically surveying Americans almost a hundred years ago.
A nation of question-askers
Wednesday, April 9, 14
Examples of some of the long-running established ways in which Americans ask each other questions. George Gallup began systematically surveying Americans almost a hundred years ago.
A nation of question-askers
Wednesday, April 9, 14
Examples of some of the long-running established ways in which Americans ask each other questions. George Gallup began systematically surveying Americans almost a hundred years ago.
A nation of question-askers
Wednesday, April 9, 14
Examples of some of the long-running established ways in which Americans ask each other questions. George Gallup began systematically surveying Americans almost a hundred years ago.
My school of question-askers
February 2013 Urban Legend
Wednesday, April 9, 14
Examples from my school, including a student cartoon that expresses frustration and fatigue over constant question-asking.
A new industry: Web-based surveys and polls
Wednesday, April 9, 14
Presidents are born ...
Wednesday, April 9, 14
... and live by polls
Poll data at nyt.com (left) and realpolitics.com (above) as accessed on on 12/18/13.Wednesday, April 9, 14
Just one day ... at Real Clear Politics.com
Wednesday, April 9, 14
Selections from the Pew Fact Tank
Wednesday, April 9, 14
Pew regularly surveys teens on technology, religion, social trends, sexuality, and more.
Making surveys sexy
Wednesday, April 9, 14
Nate Silver used his column to predict results in the 2008 and 2012 presidential elections with astonishing accuracy, correctly predicting the winner in all 50 states in 2012. Analyzing survey data is his forte. Now at ESPN, Silver says he practices data journalism.
Why surveys are powerful and scary
When they verify the popularity of an idea or proposal, surveys and polls provide persuasive
appeals because, in a democracy, majority opinion offers a compelling warrant:
A government should do what most people want.
Source: Everythings An Argument (2007: St. Martins Press)
Wednesday, April 9, 14
It always makes sense, though, to push back ... especially (when a poll) supports
your own point of view.
Wednesday, April 9, 14
Questions to ask
Have you surveyed enough people to be accurate?
Are those people representative of the selected population as a whole?
Did you choose them randomly not selecting those likely to say what you want to hear?
Does the wording of your questions intentionally or unintentionally create bias or skew results?
Have you described the results accurately and fairly? Does your story stick to what you asked, and only what you asked?
Wednesday, April 9, 14
Here are questions you should ask yourself, whether youre doing or reading a survey. Questions adapted from Arguments Based on Facts and Reason in Everythings An Argument (2007: St. Martins Press).
Three tips
Wednesday, April 9, 14
Three tips (and three deadly sins)
Wednesday, April 9, 14
Three tips (and three deadly sins) Get a random sample. It
should be representative also not just people who will answer the way you want them to.
Wednesday, April 9, 14
Three tips (and three deadly sins) Get a random sample. It
should be representative also not just people who will answer the way you want them to.
undercoverage
Wednesday, April 9, 14
Three tips (and three deadly sins) Get a random sample. It
should be representative also not just people who will answer the way you want them to.
Avoid bias. Ask a variety of questions (open/closed, multiple choice, short answer). Question order matters; begin with basics. Avoid leading questions.
undercoverage
Wednesday, April 9, 14
Three tips (and three deadly sins) Get a random sample. It
should be representative also not just people who will answer the way you want them to.
Avoid bias. Ask a variety of questions (open/closed, multiple choice, short answer). Question order matters; begin with basics. Avoid leading questions.
undercoverage
response bias
Wednesday, April 9, 14
Three tips (and three deadly sins) Get a random sample. It
should be representative also not just people who will answer the way you want
