Lies, damned lies, and surveys

Click here to load reader

  • date post

    13-Jul-2015
  • Category

    Education

  • view

    86
  • download

    1

Embed Size (px)

Transcript of Lies, damned lies, and surveys

  • Lies, damned lies ... and surveys

    Beatrice MotamediThe Urban School of San FranciscoSpring JEA/NSPA National High School Journalism ConventionApril 2014

    Wednesday, April 9, 14

  • Wednesday, April 9, 14

    Lies, damned lies, and statistics is a phrase popularized by Mark Twain, the American humorist, who wrote in Chapters from My Autobiography that "(f)igures often beguile me ... particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: 'There are three kinds of lies: lies, damned lies, and statistics.'"

  • Whats the last survey you took for your newspaper or website?

    How many people responded?

    What did you learn?

    What would you like to do better?

    Wednesday, April 9, 14

  • Lets BYOS (build your own survey)

    Find a person whose eye color is the same as yours and has the same kind of smartphone as you do (iPhone or Android)

    Brainstorm a question youd like to ask right here in San Diego. How many students surf? Who believes in global warming? Should texting while driving be a crime? Choose a topic that actually interests YOU.

    Take out your smartphone and download an app: Instasurvey (Android) or Surveymonkey (iPhone)

    Set up a free account and go out with your partner/ask your question. Try to capture at least one answer using your app. Be back in 5 minutes.

    Wednesday, April 9, 14

  • lets whip

    what did you ask?

    how did it go?

    Wednesday, April 9, 14

  • Glossary

    Poll: you ask one question, usually yes/no or multiple choice (what you just did)

    Survey: you ask several questions with various question types (multiple-choice, open/closed, short-answer essay, etc.).

    Random sample: a selection from a population, based on chance, and with an equal probability of being selected

    Respondents: those who actually respond to the survey (not the same as those you sample)

    Time/manner: when and how a survey is conducted, e.g., a Surveymonkey from April 7-10

    Response bias: a preference that results from problems in the surveyprocess, e.g., using leading questions, or not allowing answers to sensitive qs to be confidential

    Sampling error: the variation in data among samples

    Wednesday, April 9, 14

  • How not to do a survey*

    Click here for RNC Young

    Voters Survey, Stephen Colbert,

    4/4/13

    *with apologies to young Republicans in the audience (young Dems probably have a survey just as bad as this one)

    Wednesday, April 9, 14

    This video captures many typical survey mistakes, including non-response bias (you cant find the survey), response bias (poorly written or leading questions) and no control over the sample (Colbert is obviously over 30 years of age but takes the survey anyway).

  • Whats wrong here?

    Wednesday, April 9, 14

    Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.

  • Whats wrong here?

    too many qs

    Wednesday, April 9, 14

    Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.

  • Whats wrong here?

    too many qs

    too many assumptions (bias)

    Wednesday, April 9, 14

    Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.

  • Whats wrong here?

    too many qs

    too many assumptions (bias)

    closed q, not open

    Wednesday, April 9, 14

    Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.

  • Whats wrong here?

    too many qs

    too many assumptions (bias)

    closed q, not open

    you woke this guy up ... for this?

    Wednesday, April 9, 14

    Too many questions, bias, types of questions (leading) and methodology can all affect how a person answers a survey, generating results that are just plain ... wrong.

  • Our goal: A baby survey you can use right here

    Population sample WHO to ask

    One closed question WHAT to ask

    One open question ditto

    A platform HOW you will ask your questions

    Wednesday, April 9, 14

    Our goal for this presentation: You will leave with the beginnings of a survey, including a topic, two types of questions, a plan on how you will collect your sample, and a web-based platform that makes it all easier.

  • A nation of question-askers

    Wednesday, April 9, 14

    Examples of some of the long-running established ways in which Americans ask each other questions. George Gallup began systematically surveying Americans almost a hundred years ago.

  • A nation of question-askers

    Wednesday, April 9, 14

    Examples of some of the long-running established ways in which Americans ask each other questions. George Gallup began systematically surveying Americans almost a hundred years ago.

  • A nation of question-askers

    Wednesday, April 9, 14

    Examples of some of the long-running established ways in which Americans ask each other questions. George Gallup began systematically surveying Americans almost a hundred years ago.

  • A nation of question-askers

    Wednesday, April 9, 14

    Examples of some of the long-running established ways in which Americans ask each other questions. George Gallup began systematically surveying Americans almost a hundred years ago.

  • My school of question-askers

    February 2013 Urban Legend

    Wednesday, April 9, 14

    Examples from my school, including a student cartoon that expresses frustration and fatigue over constant question-asking.

  • A new industry: Web-based surveys and polls

    Wednesday, April 9, 14

  • Presidents are born ...

    Wednesday, April 9, 14

  • ... and live by polls

    Poll data at nyt.com (left) and realpolitics.com (above) as accessed on on 12/18/13.Wednesday, April 9, 14

  • Just one day ... at Real Clear Politics.com

    Wednesday, April 9, 14

  • Selections from the Pew Fact Tank

    Wednesday, April 9, 14

    Pew regularly surveys teens on technology, religion, social trends, sexuality, and more.

  • Making surveys sexy

    Wednesday, April 9, 14

    Nate Silver used his column to predict results in the 2008 and 2012 presidential elections with astonishing accuracy, correctly predicting the winner in all 50 states in 2012. Analyzing survey data is his forte. Now at ESPN, Silver says he practices data journalism.

  • Why surveys are powerful and scary

    When they verify the popularity of an idea or proposal, surveys and polls provide persuasive

    appeals because, in a democracy, majority opinion offers a compelling warrant:

    A government should do what most people want.

    Source: Everythings An Argument (2007: St. Martins Press)

    Wednesday, April 9, 14

  • It always makes sense, though, to push back ... especially (when a poll) supports

    your own point of view.

    Wednesday, April 9, 14

  • Questions to ask

    Have you surveyed enough people to be accurate?

    Are those people representative of the selected population as a whole?

    Did you choose them randomly not selecting those likely to say what you want to hear?

    Does the wording of your questions intentionally or unintentionally create bias or skew results?

    Have you described the results accurately and fairly? Does your story stick to what you asked, and only what you asked?

    Wednesday, April 9, 14

    Here are questions you should ask yourself, whether youre doing or reading a survey. Questions adapted from Arguments Based on Facts and Reason in Everythings An Argument (2007: St. Martins Press).

  • Three tips

    Wednesday, April 9, 14

  • Three tips (and three deadly sins)

    Wednesday, April 9, 14

  • Three tips (and three deadly sins) Get a random sample. It

    should be representative also not just people who will answer the way you want them to.

    Wednesday, April 9, 14

  • Three tips (and three deadly sins) Get a random sample. It

    should be representative also not just people who will answer the way you want them to.

    undercoverage

    Wednesday, April 9, 14

  • Three tips (and three deadly sins) Get a random sample. It

    should be representative also not just people who will answer the way you want them to.

    Avoid bias. Ask a variety of questions (open/closed, multiple choice, short answer). Question order matters; begin with basics. Avoid leading questions.

    undercoverage

    Wednesday, April 9, 14

  • Three tips (and three deadly sins) Get a random sample. It

    should be representative also not just people who will answer the way you want them to.

    Avoid bias. Ask a variety of questions (open/closed, multiple choice, short answer). Question order matters; begin with basics. Avoid leading questions.

    undercoverage

    response bias

    Wednesday, April 9, 14

  • Three tips (and three deadly sins) Get a random sample. It

    should be representative also not just people who will answer the way you want