1. Progressive Media Impact: Metrics, Evaluation, and Improvement Gary King Institute for Quantitative Social Science Harvard University (talk at The Media Consortium, San Francisco, 10/13/2011)Gary King (Harvard) Progressive Media Impact 1 / 11

2. Goals Gary King (Harvard) Progressive Media Impact 2 / 11

3. Goals Measure public discourse Gary King (Harvard) Progressive Media Impact 2 / 11

4. Goals Measure public discourse Estimate the causal eect of the progressive media on discourse Gary King (Harvard) Progressive Media Impact 2 / 11

5. Goals Measure public discourse Estimate the causal eect of the progressive media on discourse Learn together how to make the progressive media more eective Gary King (Harvard) Progressive Media Impact 2 / 11

6. Goals Measure public discourse Estimate the causal eect of the progressive media on discourse Learn together how to make the progressive media more eective Take advantage of recent dramatic scientic advances: Gary King (Harvard) Progressive Media Impact 2 / 11

7. Goals Measure public discourse Estimate the causal eect of the progressive media on discourse Learn together how to make the progressive media more eective Take advantage of recent dramatic scientic advances: Informative data now available on the public debate Gary King (Harvard) Progressive Media Impact 2 / 11

8. Goals Measure public discourse Estimate the causal eect of the progressive media on discourse Learn together how to make the progressive media more eective Take advantage of recent dramatic scientic advances: Informative data now available on the public debate Measurement methods: meaning, not word counts Gary King (Harvard) Progressive Media Impact 2 / 11

9. Goals Measure public discourse Estimate the causal eect of the progressive media on discourse Learn together how to make the progressive media more eective Take advantage of recent dramatic scientic advances: Informative data now available on the public debate Measurement methods: meaning, not word counts Experimental designs: avoiding confounding factors Gary King (Harvard) Progressive Media Impact 2 / 11

10. Goals Measure public discourse Estimate the causal eect of the progressive media on discourse Learn together how to make the progressive media more eective Take advantage of recent dramatic scientic advances: Informative data now available on the public debate Measurement methods: meaning, not word counts Experimental designs: avoiding confounding factors Causal eect estimation: less bias & ineciency Gary King (Harvard) Progressive Media Impact 2 / 11

11. Goals Measure public discourse Estimate the causal eect of the progressive media on discourse Learn together how to make the progressive media more eective Take advantage of recent dramatic scientic advances: Informative data now available on the public debate Measurement methods: meaning, not word counts Experimental designs: avoiding confounding factors Causal eect estimation: less bias & ineciency New ways of working together to improve impact Gary King (Harvard) Progressive Media Impact 2 / 11

12. Who Frames the Public Debate? Gary King (Harvard) Progressive Media Impact 3 / 11

13. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? Gary King (Harvard) Progressive Media Impact 3 / 11

14. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. Gary King (Harvard) Progressive Media Impact 3 / 11

15. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. The next day which the president spent in hiding 90% approved. Gary King (Harvard) Progressive Media Impact 3 / 11

16. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. The next day which the president spent in hiding 90% approved. Was this massive opinion change, or did the 9/11 frame change how we viewed the question? Gary King (Harvard) Progressive Media Impact 3 / 11

17. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. The next day which the president spent in hiding 90% approved. Was this massive opinion change, or did the 9/11 frame change how we viewed the question? The frame: imposed by events, not the media Gary King (Harvard) Progressive Media Impact 3 / 11

18. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. The next day which the president spent in hiding 90% approved. Was this massive opinion change, or did the 9/11 frame change how we viewed the question? The frame: imposed by events, not the media Public opinion polls: measuring what? Gary King (Harvard) Progressive Media Impact 3 / 11

19. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. The next day which the president spent in hiding 90% approved. Was this massive opinion change, or did the 9/11 frame change how we viewed the question? The frame: imposed by events, not the media Public opinion polls: measuring what?Support for Ban on Partial Birth Abortion Gary King (Harvard) Progressive Media Impact 3 / 11

20. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. The next day which the president spent in hiding 90% approved. Was this massive opinion change, or did the 9/11 frame change how we viewed the question? The frame: imposed by events, not the media Public opinion polls: measuring what?Support for Ban on Partial Birth Abortion Supporters: use baby Gary King (Harvard) Progressive Media Impact 3 / 11

21. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. The next day which the president spent in hiding 90% approved. Was this massive opinion change, or did the 9/11 frame change how we viewed the question? The frame: imposed by events, not the media Public opinion polls: measuring what?Support for Ban on Partial Birth Abortion Supporters: use baby Opponents: use fetus Gary King (Harvard) Progressive Media Impact 3 / 11

22. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. The next day which the president spent in hiding 90% approved. Was this massive opinion change, or did the 9/11 frame change how we viewed the question? The frame: imposed by events, not the media Public opinion polls: measuring what?Support for Ban on Partial Birth Abortion Supporters: use baby Opponents: use fetus In surveys, baby increases support for the ban by 30%! Gary King (Harvard) Progressive Media Impact 3 / 11

23. Who Frames the Public Debate?Do you approve of how George W. Bush is handling his job? On 9/10/2001, 55% of Americans approved of the way George W. Bush was handling his job as president. The next day which the president spent in hiding 90% approved. Was this massive opinion change, or did the 9/11 frame change how we viewed the question? The frame: imposed by events, not the media Public opinion polls: measuring what?Support for Ban on Partial Birth Abortion Supporters: use baby Opponents: use fetus In surveys, baby increases support for the ban by 30%! Control the frame & you control the debate and policy outcome Gary King (Harvard) Progressive Media Impact 3 / 11

24. Partial Birth Abortion Ban: Who controls the frame? Gary King (Harvard) Progressive Media Impact 4 / 11

25. indicate their level of support for a PBA ban). Figure 1 shows that aggregate supportPartial Birth Abortion Ban: Who controls the frame? for a ban seemed to rise and fall in tandem with the proportion of baby usage in news stories about PBA. Stenberg v.Carhart 1 70 argued 0.8 Second Clinton Proportion veto Support First 0.6 Clinton veto 60 0.4 0.2 0 50 1995 1996 1997 1998 1999 2000 Proportion "Baby" in Media Support (PSRA) Trend Line (20 period moving average) Support (Gallup) Figure 1 Relationship between media discourse and public support for partial-birth abor- tion ban. Note: The media series represents a 5-week moving average of the proportion of baby mentions. Public opinion data come from similarly worded questions in surveys by Gallup (squares) and Princeton Survey Research Associates (triangles). These items ask respondents about their level of support for a ban on partial-birth abortion. Question wording can be obtained from LexisNexis or the iPoll database at the Roper Center for Public Opinion. Gary King (Harvard) Progressive Media Impact 4 / 11

26. indicate their level of support for a PBA ban). Figure 1 shows that aggregate supportPartial Birth Abortion Ban: Who controls the frame? for a ban seemed to rise and fall in tandem with the proportion of baby usage in news stories about PBA. Stenberg v.Carhart 1 70 argued 0.8 Second Clinton Proportion veto Support First 0.6 Clinton veto 60 0.4 0.2 0 50 1995 1996 1997 1998 1999 2000 Proportion "Baby" in Media Support (PSRA) Trend Line (20 period moving average) Support (Gallup) Figure 1 Relationship between media discourse and public support for partial-birth abor- tion ban. When baby is used in the media, support forthe proportion increases Note: The media series represents a 5-week moving average of PBA ban of baby mentions. Public opinion data come from similarly worded questions in surveys by Gallup (squares) and Princeton Survey Research Associates (triangles). These items ask respondents about their level of support for a ban on partial-birth abortion. Question wording can be obtained from LexisNexis or the iPoll database at the Roper Center for Public Opinion. Gary King (Harvard) Progressive Media Impact 4 / 11

27. indicate their level of support for a PBA ban). Figure 1 shows that aggregate supportPartial Birth Abortion Ban: Who controls the frame? for a ban seemed to rise and fall in tandem with the proportion of baby usage in news stories about PBA. Stenberg v.Carhart 1 70 argued 0.8 Second Clinton Proportion veto Support First 0.6 Clinton veto 60 0.4 0.2 0 50 1995 1996 1997 1998 1999 2000 Proportion "Baby" in Media Support (PSRA) Trend Line (20 period moving average) Support (Gallup) Figure 1 Relationship between media discourse and public support for partial-birth abor- tion ban. When baby is used in the media, support forthe proportion increases Note: The media series represents a 5-week moving average of PBA ban of baby Did the Public opinion data come from similarly wordeddid public surveys by Gallup mentions. media inuence public support or questions in support inuence the media? of support forResearch Associates (triangles). These items ask respondents (squares) and Princeton Survey about their level a ban on partial-birth abortion. Question wording can be obtained from LexisNexis or the iPoll database at the Roper Center for Public Opinion. Gary King (Harvard) Progressive Media Impact 4 / 11

28. indicate their level of support for a PBA ban). Figure 1 shows that aggregate supportPartial Birth Abortion Ban: Who controls the frame? for a ban seemed to rise and fall in tandem with the proportion of baby usage in news stories about PBA. Stenberg v.Carhart 1 70 argued 0.8 Second Clinton Proportion veto Support First 0.6 Clinton veto 60 0.4 0.2 0 50 1995 1996 1997 1998 1999 2000 Proportion "Baby" in Media Support (PSRA) Trend Line (20 period moving average) Support (Gallup) Figure 1 Relationship between media discourse and public support for partial-birth abor- tion ban. When baby is used in the media, support forthe proportion increases Note: The media series represents a 5-week moving average of PBA ban of baby Did the Public opinion data come from similarly wordeddid public surveys by Gallup mentions. media inuence public support or questions in support inuence the media? How canforResearch Associates (triangles). These items ask respondents (squares) and Princeton Survey about their level of support we ban on partial-birth abortion. Question wording can be a tell? obtained from LexisNexis or the iPoll database at the Roper Center for Public Opinion. Gary King (Harvard) Progressive Media Impact 4 / 11

29. indicate their level of support for a PBA ban). Figure 1 shows that aggregate supportPartial Birth Abortion Ban: Who controls the frame? for a ban seemed to rise and fall in tandem with the proportion of baby usage in news stories about PBA. Stenberg v.Carhart 1 70 argued 0.8 Second Clinton Proportion veto Support First 0.6 Clinton veto 60 0.4 0.2 0 50 1995 1996 1997 1998 1999 2000 Proportion "Baby" in Media Support (PSRA) Trend Line (20 period moving average) Support (Gallup) Figure 1 Relationship between media discourse and public support for partial-birth abor- tion ban. When baby is used in the media, support forthe proportion increases Note: The media series represents a 5-week moving average of PBA ban of baby Did the Public opinion data come from similarly wordeddid public surveys by Gallup mentions. media inuence public support or questions in support inuence the media? How canforResearch Associates (triangles). These items ask respondents (squares) and Princeton Survey about their level of support we ban on partial-birth abortion. Question wording can be a tell? Do babyLexisNexis or the iPoll database at the Roper Center for Public Opinion. obtained from and fetus word counts even measure what we intend? Gary King (Harvard) Progressive Media Impact 4 / 11

30. The Plan, 1: Collect New Data Gary King (Harvard) Progressive Media Impact 5 / 11

31. The Plan, 1: Collect New DataData on Public Discourse Gary King (Harvard) Progressive Media Impact 5 / 11

32. The Plan, 1: Collect New DataData on Public Discourse Until recently, measures of public discourse were inadequate: public opinion polls, content analyses of newspaper editorials; systematic, real time, informative data nonexistent Gary King (Harvard) Progressive Media Impact 5 / 11

33. The Plan, 1: Collect New DataData on Public Discourse Until recently, measures of public discourse were inadequate: public opinion polls, content analyses of newspaper editorials; systematic, real time, informative data nonexistent Now: Gary King (Harvard) Progressive Media Impact 5 / 11

34. The Plan, 1: Collect New DataData on Public Discourse Until recently, measures of public discourse were inadequate: public opinion polls, content analyses of newspaper editorials; systematic, real time, informative data nonexistent Now: Hallway conversations appear in the 1.4B social media posts a week Gary King (Harvard) Progressive Media Impact 5 / 11

35. The Plan, 1: Collect New DataData on Public Discourse Until recently, measures of public discourse were inadequate: public opinion polls, content analyses of newspaper editorials; systematic, real time, informative data nonexistent Now: Hallway conversations appear in the 1.4B social media posts a week Growing sources: blogs, Twitter, Facebook, forums, chat rooms, etc. Gary King (Harvard) Progressive Media Impact 5 / 11

36. The Plan, 1: Collect New DataData on Public Discourse Until recently, measures of public discourse were inadequate: public opinion polls, content analyses of newspaper editorials; systematic, real time, informative data nonexistent Now: Hallway conversations appear in the 1.4B social media posts a week Growing sources: blogs, Twitter, Facebook, forums, chat rooms, etc.Data on Media OutletsWork together to nd or create Gary King (Harvard) Progressive Media Impact 5 / 11

37. The Plan, 1: Collect New DataData on Public Discourse Until recently, measures of public discourse were inadequate: public opinion polls, content analyses of newspaper editorials; systematic, real time, informative data nonexistent Now: Hallway conversations appear in the 1.4B social media posts a week Growing sources: blogs, Twitter, Facebook, forums, chat rooms, etc.Data on Media OutletsWork together to nd or create Existing data on outputs (web trac, donations, comments, emails, phone logs, etc.) Gary King (Harvard) Progressive Media Impact 5 / 11

38. The Plan, 1: Collect New DataData on Public Discourse Until recently, measures of public discourse were inadequate: public opinion polls, content analyses of newspaper editorials; systematic, real time, informative data nonexistent Now: Hallway conversations appear in the 1.4B social media posts a week Growing sources: blogs, Twitter, Facebook, forums, chat rooms, etc.Data on Media OutletsWork together to nd or create Existing data on outputs (web trac, donations, comments, emails, phone logs, etc.) Existing data on content: e.g., tagged story databases, web artifacts, listener information, etc. Gary King (Harvard) Progressive Media Impact 5 / 11

39. The Plan, 1: Collect New DataData on Public Discourse Until recently, measures of public discourse were inadequate: public opinion polls, content analyses of newspaper editorials; systematic, real time, informative data nonexistent Now: Hallway conversations appear in the 1.4B social media posts a week Growing sources: blogs, Twitter, Facebook, forums, chat rooms, etc.Data on Media OutletsWork together to nd or create Existing data on outputs (web trac, donations, comments, emails, phone logs, etc.) Existing data on content: e.g., tagged story databases, web artifacts, listener information, etc. New systematic data from texts of stories, audio-to-text, etc. Gary King (Harvard) Progressive Media Impact 5 / 11

40. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Gary King (Harvard) Progressive Media Impact 6 / 11

41. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Gary King (Harvard) Progressive Media Impact 6 / 11

42. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Misleading word counts Gary King (Harvard) Progressive Media Impact 6 / 11

43. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Misleading word counts Impossible amounts of reading Gary King (Harvard) Progressive Media Impact 6 / 11

44. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Misleading word counts Impossible amounts of reading Now, new statistical methods for text analytics: Gary King (Harvard) Progressive Media Impact 6 / 11

45. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Misleading word counts Impossible amounts of reading Now, new statistical methods for text analytics: Accurately summarize huge volumes of information Gary King (Harvard) Progressive Media Impact 6 / 11

46. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Misleading word counts Impossible amounts of reading Now, new statistical methods for text analytics: Accurately summarize huge volumes of information Better than humans alone, or computers alone, we amplify human intelligence Gary King (Harvard) Progressive Media Impact 6 / 11

47. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Misleading word counts Impossible amounts of reading Now, new statistical methods for text analytics: Accurately summarize huge volumes of information Better than humans alone, or computers alone, we amplify human intelligence Works fast, in near real time Gary King (Harvard) Progressive Media Impact 6 / 11

48. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Misleading word counts Impossible amounts of reading Now, new statistical methods for text analytics: Accurately summarize huge volumes of information Better than humans alone, or computers alone, we amplify human intelligence Works fast, in near real time Can measure media frames (about politicians or policies) by volume, sentiment, topics, perspectives or any categories we think of Gary King (Harvard) Progressive Media Impact 6 / 11

49. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Misleading word counts Impossible amounts of reading Now, new statistical methods for text analytics: Accurately summarize huge volumes of information Better than humans alone, or computers alone, we amplify human intelligence Works fast, in near real time Can measure media frames (about politicians or policies) by volume, sentiment, topics, perspectives or any categories we think of Measure as far back in time as feasible (6-24 months?) Gary King (Harvard) Progressive Media Impact 6 / 11

50. The Plan, 2: Adapt New Methods to Understand Billionsof Social Media Posts, & News Stories Until recently, methods were inadequate: Misleading word counts Impossible amounts of reading Now, new statistical methods for text analytics: Accurately summarize huge volumes of information Better than humans alone, or computers alone, we amplify human intelligence Works fast, in near real time Can measure media frames (about politicians or policies) by volume, sentiment, topics, perspectives or any categories we think of Measure as far back in time as feasible (6-24 months?) Design systematic measurement going forward Gary King (Harvard) Progressive Media Impact 6 / 11

51. [Some detail on innovations in text analytics] Gary King (Harvard) Progressive Media Impact 7 / 11

52. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Gary King (Harvard) Progressive Media Impact 7 / 11

53. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Gary King (Harvard) Progressive Media Impact 7 / 11

54. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Gary King (Harvard) Progressive Media Impact 7 / 11

55. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is Gary King (Harvard) Progressive Media Impact 7 / 11

56. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit Gary King (Harvard) Progressive Media Impact 7 / 11

57. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit, (2) an attack on public sector workers Gary King (Harvard) Progressive Media Impact 7 / 11

58. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit, (2) an attack on public sector workers, (3) an attack on the progressive movement generally Gary King (Harvard) Progressive Media Impact 7 / 11

59. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit, (2) an attack on public sector workers, (3) an attack on the progressive movement generally, or (4) due to an arrogant publicity-hungry Governor Gary King (Harvard) Progressive Media Impact 7 / 11

60. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit, (2) an attack on public sector workers, (3) an attack on the progressive movement generally, or (4) due to an arrogant publicity-hungry Governor How to sort billions of social media posts into categories? Gary King (Harvard) Progressive Media Impact 7 / 11

61. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit, (2) an attack on public sector workers, (3) an attack on the progressive movement generally, or (4) due to an arrogant publicity-hungry Governor How to sort billions of social media posts into categories? Classify by hand: infeasible for more than a sample at one point in time Gary King (Harvard) Progressive Media Impact 7 / 11

62. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit, (2) an attack on public sector workers, (3) an attack on the progressive movement generally, or (4) due to an arrogant publicity-hungry Governor How to sort billions of social media posts into categories? Classify by hand: infeasible for more than a sample at one point in time Guess what words imply which categories: inaccurate Gary King (Harvard) Progressive Media Impact 7 / 11

63. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit, (2) an attack on public sector workers, (3) an attack on the progressive movement generally, or (4) due to an arrogant publicity-hungry Governor How to sort billions of social media posts into categories? Classify by hand: infeasible for more than a sample at one point in time Guess what words imply which categories: inaccurate Use machine learning methods with hand-coded training set to automatically classify: at best 60-70% accuracy (useful for Google searches, useless for category percentages) Gary King (Harvard) Progressive Media Impact 7 / 11

64. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit, (2) an attack on public sector workers, (3) an attack on the progressive movement generally, or (4) due to an arrogant publicity-hungry Governor How to sort billions of social media posts into categories? Classify by hand: infeasible for more than a sample at one point in time Guess what words imply which categories: inaccurate Use machine learning methods with hand-coded training set to automatically classify: at best 60-70% accuracy (useful for Google searches, useless for category percentages) Use new statistical method with hand-coded training set to estimate category percentages (without individual classications): Extremely accurate Gary King (Harvard) Progressive Media Impact 7 / 11

65. [Some detail on innovations in text analytics] How can humans understand large numbers of social media posts (or news stories)? Read & interpret: infeasible Sort into a few categories; track category percentages over time Example of categories: Wisconsin budget stando is (1) a responsible eort to x budget decit, (2) an attack on public sector workers, (3) an attack on the progressive movement generally, or (4) due to an arrogant publicity-hungry Governor How to sort billions of social media posts into categories? Classify by hand: infeasible for more than a sample at one point in time Guess what words imply which categories: inaccurate Use machine learning methods with hand-coded training set to automatically classify: at best 60-70% accuracy (useful for Google searches, useless for category percentages) Use new statistical method with hand-coded training set to estimate category percentages (without individual classications): Extremely accurate Finding the needle in the haystack (Google search) = characterizing the haystack, and so new methods were required Gary King (Harvard) Progressive Media Impact 7 / 11

66. The Plan, 3: Use New Methods of Causal Inference forObservational Data Gary King (Harvard) Progressive Media Impact 8 / 11

67. The Plan, 3: Use New Methods of Causal Inference forObservational Data Until recently: Estimating causal eects from observational data was very risky and often illusory Gary King (Harvard) Progressive Media Impact 8 / 11

68. The Plan, 3: Use New Methods of Causal Inference forObservational Data Until recently: Estimating causal eects from observational data was very risky and often illusory Now, new methods: Gary King (Harvard) Progressive Media Impact 8 / 11

69. The Plan, 3: Use New Methods of Causal Inference forObservational Data Until recently: Estimating causal eects from observational data was very risky and often illusory Now, new methods: Give strong hints about causal relationships Gary King (Harvard) Progressive Media Impact 8 / 11

70. The Plan, 3: Use New Methods of Causal Inference forObservational Data Until recently: Estimating causal eects from observational data was very risky and often illusory Now, new methods: Give strong hints about causal relationships Reveal all hidden assumptions (ignorability, interference, etc.) Gary King (Harvard) Progressive Media Impact 8 / 11

71. The Plan, 3: Use New Methods of Causal Inference forObservational Data Until recently: Estimating causal eects from observational data was very risky and often illusory Now, new methods: Give strong hints about causal relationships Reveal all hidden assumptions (ignorability, interference, etc.) Help inform what experiments to run Gary King (Harvard) Progressive Media Impact 8 / 11

72. The Plan, 3: Use New Methods of Causal Inference forObservational Data Until recently: Estimating causal eects from observational data was very risky and often illusory Now, new methods: Give strong hints about causal relationships Reveal all hidden assumptions (ignorability, interference, etc.) Help inform what experiments to run Well use new methods to: Gary King (Harvard) Progressive Media Impact 8 / 11

73. The Plan, 3: Use New Methods of Causal Inference forObservational Data Until recently: Estimating causal eects from observational data was very risky and often illusory Now, new methods: Give strong hints about causal relationships Reveal all hidden assumptions (ignorability, interference, etc.) Help inform what experiments to run Well use new methods to: Estimate the eect of progressive media on use of media frames, & sentiment about them, in public discourse Gary King (Harvard) Progressive Media Impact 8 / 11

74. The Plan, 3: Use New Methods of Causal Inference forObservational Data Until recently: Estimating causal eects from observational data was very risky and often illusory Now, new methods: Give strong hints about causal relationships Reveal all hidden assumptions (ignorability, interference, etc.) Help inform what experiments to run Well use new methods to: Estimate the eect of progressive media on use of media frames, & sentiment about them, in public discourse Develop strong testable hypotheses about what do next Gary King (Harvard) Progressive Media Impact 8 / 11

75. The Plan, 3: Use New Methods of Causal Inference forObservational Data Until recently: Estimating causal eects from observational data was very risky and often illusory Now, new methods: Give strong hints about causal relationships Reveal all hidden assumptions (ignorability, interference, etc.) Help inform what experiments to run Well use new methods to: Estimate the eect of progressive media on use of media frames, & sentiment about them, in public discourse Develop strong testable hypotheses about what do next Well do everything that can be done this way before turning to experiments (and using your time!) Gary King (Harvard) Progressive Media Impact 8 / 11

76. The Plan, 4: Use New Experimental Designs Gary King (Harvard) Progressive Media Impact 9 / 11

77. The Plan, 4: Use New Experimental Designs Observational analysis: some evidence of what might have worked in the past Gary King (Harvard) Progressive Media Impact 9 / 11

78. The Plan, 4: Use New Experimental Designs Observational analysis: some evidence of what might have worked in the past Experiments: gold standard for evaluating causal claims Gary King (Harvard) Progressive Media Impact 9 / 11

79. The Plan, 4: Use New Experimental Designs Observational analysis: some evidence of what might have worked in the past Experiments: gold standard for evaluating causal claims Experiments benets from: investigator control (usually randomized) over the treatment (drugs, stories) Gary King (Harvard) Progressive Media Impact 9 / 11

80. The Plan, 4: Use New Experimental Designs Observational analysis: some evidence of what might have worked in the past Experiments: gold standard for evaluating causal claims Experiments benets from: investigator control (usually randomized) over the treatment (drugs, stories) What randomized treatments get us: a variable unrelated to all confounders Gary King (Harvard) Progressive Media Impact 9 / 11

81. The Plan, 4: Use New Experimental Designs Observational analysis: some evidence of what might have worked in the past Experiments: gold standard for evaluating causal claims Experiments benets from: investigator control (usually randomized) over the treatment (drugs, stories) What randomized treatments get us: a variable unrelated to all confounders How experiments fail: Gary King (Harvard) Progressive Media Impact 9 / 11

82. The Plan, 4: Use New Experimental Designs Observational analysis: some evidence of what might have worked in the past Experiments: gold standard for evaluating causal claims Experiments benets from: investigator control (usually randomized) over the treatment (drugs, stories) What randomized treatments get us: a variable unrelated to all confounders How experiments fail: Insensitivity to research subjects, political interests, or local context Gary King (Harvard) Progressive Media Impact 9 / 11

83. The Plan, 4: Use New Experimental Designs Observational analysis: some evidence of what might have worked in the past Experiments: gold standard for evaluating causal claims Experiments benets from: investigator control (usually randomized) over the treatment (drugs, stories) What randomized treatments get us: a variable unrelated to all confounders How experiments fail: Insensitivity to research subjects, political interests, or local context Examples: Drug trials; Mexicos Anti-Poverty program Gary King (Harvard) Progressive Media Impact 9 / 11

84. The Plan, 4: Use New Experimental Designs Observational analysis: some evidence of what might have worked in the past Experiments: gold standard for evaluating causal claims Experiments benets from: investigator control (usually randomized) over the treatment (drugs, stories) What randomized treatments get us: a variable unrelated to all confounders How experiments fail: Insensitivity to research subjects, political interests, or local context Examples: Drug trials; Mexicos Anti-Poverty program solutions always exist; well nd them together Gary King (Harvard) Progressive Media Impact 9 / 11

85. The Plan, 4: Use New Experimental Designs Observational analysis: some evidence of what might have worked in the past Experiments: gold standard for evaluating causal claims Experiments benets from: investigator control (usually randomized) over the treatment (drugs, stories) What randomized treatments get us: a variable unrelated to all confounders How experiments fail: Insensitivity to research subjects, political interests, or local context Examples: Drug trials; Mexicos Anti-Poverty program solutions always exist; well nd them together New experimental designs: let randomization survive disruption Gary King (Harvard) Progressive Media Impact 9 / 11

86. The Plan, 5: Implementation Strategy Gary King (Harvard) Progressive Media Impact 10 / 11

87. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Gary King (Harvard) Progressive Media Impact 10 / 11

88. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Oer optional small grants to media outlets to encourage covering particular stories (everyone will get a chance, no requirements, all decision making over content remains with organization) Gary King (Harvard) Progressive Media Impact 10 / 11

89. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Oer optional small grants to media outlets to encourage covering particular stories (everyone will get a chance, no requirements, all decision making over content remains with organization) Find otherwise arbitrary decisions to randomize (e.g., a story were indierent to being above or below the fold; the order of stories to present) Gary King (Harvard) Progressive Media Impact 10 / 11

90. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Oer optional small grants to media outlets to encourage covering particular stories (everyone will get a chance, no requirements, all decision making over content remains with organization) Find otherwise arbitrary decisions to randomize (e.g., a story were indierent to being above or below the fold; the order of stories to present) Example: Mexicos Health Insurance evaluation Gary King (Harvard) Progressive Media Impact 10 / 11

91. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Oer optional small grants to media outlets to encourage covering particular stories (everyone will get a chance, no requirements, all decision making over content remains with organization) Find otherwise arbitrary decisions to randomize (e.g., a story were indierent to being above or below the fold; the order of stories to present) Example: Mexicos Health Insurance evaluation Our scarcest resource: your time (so observational work rst) Gary King (Harvard) Progressive Media Impact 10 / 11

92. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Oer optional small grants to media outlets to encourage covering particular stories (everyone will get a chance, no requirements, all decision making over content remains with organization) Find otherwise arbitrary decisions to randomize (e.g., a story were indierent to being above or below the fold; the order of stories to present) Example: Mexicos Health Insurance evaluation Our scarcest resource: your time (so observational work rst) Experiments will Gary King (Harvard) Progressive Media Impact 10 / 11

93. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Oer optional small grants to media outlets to encourage covering particular stories (everyone will get a chance, no requirements, all decision making over content remains with organization) Find otherwise arbitrary decisions to randomize (e.g., a story were indierent to being above or below the fold; the order of stories to present) Example: Mexicos Health Insurance evaluation Our scarcest resource: your time (so observational work rst) Experiments will start small (probably individual media outlets) Gary King (Harvard) Progressive Media Impact 10 / 11

94. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Oer optional small grants to media outlets to encourage covering particular stories (everyone will get a chance, no requirements, all decision making over content remains with organization) Find otherwise arbitrary decisions to randomize (e.g., a story were indierent to being above or below the fold; the order of stories to present) Example: Mexicos Health Insurance evaluation Our scarcest resource: your time (so observational work rst) Experiments will start small (probably individual media outlets), be sequential (each will build on the previous) Gary King (Harvard) Progressive Media Impact 10 / 11

95. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Oer optional small grants to media outlets to encourage covering particular stories (everyone will get a chance, no requirements, all decision making over content remains with organization) Find otherwise arbitrary decisions to randomize (e.g., a story were indierent to being above or below the fold; the order of stories to present) Example: Mexicos Health Insurance evaluation Our scarcest resource: your time (so observational work rst) Experiments will start small (probably individual media outlets), be sequential (each will build on the previous), and grow in extent (larger experiments, with more outlets, as we learn more) Gary King (Harvard) Progressive Media Impact 10 / 11

96. The Plan, 5: Implementation Strategy Examples of how to experiment without infringing on editorial discretion: Oer optional small grants to media outlets to encourage covering particular stories (everyone will get a chance, no requirements, all decision making over content remains with organization) Find otherwise arbitrary decisions to randomize (e.g., a story were indierent to being above or below the fold; the order of stories to present) Example: Mexicos Health Insurance evaluation Our scarcest resource: your time (so observational work rst) Experiments will start small (probably individual media outlets), be sequential (each will build on the previous), and grow in extent (larger experiments, with more outlets, as we learn more) Our goal: Turn evidence of what has worked into increasingly eective strategies Gary King (Harvard) Progressive Media Impact 10 / 11

97. For More Information GKing.Harvard.edu Gary King (Harvard) Progressive Media Impact 11 / 11