Forget Big Data - Measure What...
Transcript of Forget Big Data - Measure What...
Forget Big Data - Measure What Matters
Katie Delahaye Paine
CEO
Paine Publishing
www.painepublishing.com
@queenofmetrics
Paine Publishing: Providing communications
professionals the knowledge and information they need to navigate the journey to good measurement
Newsletters
Training Courses
Consulting painepublishing.com
Katie Delahaye Paine: Helping communications professionals define and measure success for 25 years.
Founder of: The Delahaye Group KDPaine & Partners Paine Publishing
Author of: Measuring the Networked
Non-Profit Measure What Matters Measuring Public
Relationships [email protected]
Don’t Measure What’s Easy
3
Cartoon by Rob Cottingham
Measure What Matters
10/11/2014 News Group International 4
Cartoon by Rob Cottingham
Welcome to the, “I want what
I want, not what you send
me,” era
aka: “The Relationship Era”
®
It’s the end of the world as we knew it
The average audience for a MyDrunkKitchen video (http://www.youtube.com/watch?v=uSXQNred3is
1,000,000
CNN average nightly audience 179,000
The number of times per hour Digital Natives switch media—every 2.2
minutes. 27
The percent of conversation that happens OFF LINE 90%
The amount of conversations generated by bots, spammers and pay-
per-click sites 40%
The percent of on-line conversations that are public 10%
The percent of Facebook & Twitter posts that are actually seen < 5%
We need a new Attribution Model
ROI
Other Paid
Marketing
Digital/On-line
Media Buy
Print/TV Media
Buy
ROI
Media Relations
/Social Media/PR
Digital/ Online Media
Buy
Print/TV
Media Buy
Media
Relations/
Social
Media/PR
What’s Changed?
Collapse of mass media
Growth of media
everywhere
Intolerance for messaging
The Barcelona Principles
8
“Viewers are more likely to stop watching commercials at the moment in which brand logos appear on the screen” - ARF Study
The Barcelona Principles, The Conclave &
Industry Standards
1. Importance of Goal Setting and Measurement
2. Measuring the Effect on Outcomes is Preferred to Measuring
Outputs
3. The Effect on Business Results Can and Should Be Measured
Where Possible
4. Media Measurement Requires Quantity and Quality
5. Earned Media Value/AVEs are not the value of Public Relations
6. Social Media Can and Should be Measured
7. Transparency and Replicability are Paramount to Sound
Measurement
All standards are available on
http://www.painepublishing.com/st
andards-central
The Conclave
Social Media Measurement Standards have been published for:
Content Sourcing & Methods 1
Reach and Impressions 2
Engagement 3
Influence & Relevance 4
Opinion & Advocacy 5
Impact & Value 6
11/10/2014
Eyeball counting
HITS Outcomes
MSM Online Social Media
Impressions are not awareness. Where’s the “So What”?
11
Likes Are Not Engagement
12
Advocacy Commitment Trial/Consideration
Followers Likes Impressions
11/10/2014
The Engagement Process
Listen
Purchase
Run Achieve Goal
Donate
Measuring the Engagement Process
Web Analytics
eCommerce
iCloud &
Google+
Map My Run/Social Sharing
CRM/Convio
#3: Standards for Engagement
Engagement is an action that happens in response to content – i.e. when someone
engages with you. (not “about you”)
Engagement manifests differently by channel, but is typically measurable based on effort
required and how it is shared with others.
Engagement can be desirable or undesirable
Any measure of engagement must be tied to the goals and objectives for your program
Engagement occurs both off and online, and both must be considered if you intend to
integrate your metrics with other marketing or communications efforts.
Engagement includes such actions as likes, +1, shares, votes, comments, links, links,
retweets, Facebook's “Talking about you” etc.
Engagement actions should be counted by % of the audience that responds.
968 1027
914
208
297 261 269
215 222 145
328
204
510
188 222 234
41 71 50 36 25 56 30 71 45
107
Share & Likes Over Time
Average of PostLikes Average of PShares
GWA’s, Photos Drive Shares & Likes
Mia Farrow
David Beckham & Dierks Bentley
Video
• Influence happens when you are persuaded to change behavior or opinion that
would otherwise not have changed.
• Influence can be attributed to either an outlet or an individual.
• Influence happens online and off line and both should be considered
• Should include some combination of the following five elements:
• Reach
• Engagement around individual
• Relevance to topic
• Frequency of posts around the topic
• Audience impact as measured by the ability to get the target audience to
change behavior or opinion.
• If an individual scores a 0 on one element, they have no influence
• Influence cannot be expressed in a single score or algorithm
#4: Influence & Relevance
Big Numbers Influence
All influence is relative
Who is influential depends on your audience
A computer cannot tell you who matters most
To be influential requires relevance, frequency & reach
#5 Standards for Measuring Tone/Sentiment
Whatever process is defined and applied, it must be used
consistently throughout any analysis.
Sentiment reliability varies by vendor and approach – therefore
coding definitions, consistency and transparency are critical.
Opinions, recommendations and other qualitative measures are
typically more valuable than raw sentiment and increasingly
measurable:
Opinions (“it’s a good product”)
Recommendations (“try it” or “avoid it”)
Feeling/Emotions (“That product makes me feel happy”)
Intended action (“I’m going to buy that product tomorrow”)
Testing the accuracy of coding
48.94%
68.31%
88.64%
0.00% 20.00% 40.00% 60.00% 80.00% 100.00%
SDL
UberVu
NetBase
% Agreement with human coding
SDL UberVu NetBase
33.12%
58.00%
7.84%
11.95%
26.53%
13.00%
0.32%
9.46%
40.35%
30.00%
92.11%
78.00%
SDL
Beyond
NetBase
UberVu
% positive %negative %neutral
Everyone agrees on Neutral
Not so with Negative and Positive
100.00%
62.32%
14.82%
72.36%
0.00%
37.68%
85.18%
27.64%
Mixed
Negative
Neutral
Positive
Degree to which agreement occurred
No Yes
• Impact and value will always be dependent on client objectives
• Need to define outcomes in advance – will likely span multiple business
goals, especially for social (crosses disciplines)
• “ROI” should be strictly limited to measurable financial impact; “total
value” can be used for financial and non-financial impact combination
• Value can be calculated in positive returns (sales, reputation, etc.) or
avoided negative returns (risk mitigated, costs avoided)
• Key performance indicators and balanced scorecards are helpful to
connect social media impact to business results/language
#6: Impact & Value
ROI
ROI is:
Cost savings
Greater efficiency
Lower legal costs
Faster time to implementation
Lower cost per impact vs. other
marketing channels
Lower cost per message
exposure vs. other channels
ROI is not:
Impressions
Earned media value or AVE
Number of shares, views or
likes
Conversations
$$$$$$$$$$$$$$$$$$$$$$$$$$
Savings, shorter cycles, more
renewals, better ideas,
Product
Support HR IR Sales CI R&D Mktg
Research
Prod.
Mktg
PR//Mkt
24
Social Media isn’t media, it’s your business
11/10/2014
Good Relationships Are More Cost Effective
Type I love Zappos into Google, and you find
1.19 million references
Type Citibank and you get 21,000 references.
Citibank spends 100 times more a year on
advertising than Zappos.
Cost per delegate acquired:
Obama: $6,024
Clinton: $147,058
Romney: $2,389,464
So What?
It’s not about the media, it’s about
your business and your customer
It’s not how loud you’re shouting, it’s
what your stakeholders do after
they’ve heard your message and your
relationships.
There are no boundaries
It’s not about how big your data is, it’s
about how you use it
6 steps to standards-compliant
measurement
Step 1: Define your goal(s).
What outcomes is this strategy or tactic going to achieve?
What are your measurable objectives?
Step 2: Define the parameters
Who are you are trying to reach? How do your efforts
connect with those audiences to achieve the goal.
Step 3: Define your benchmarks.
Who or what are you going to compare your results to?
Step 4: Define your metrics.
What are the indicators to judge your progress?
Step 5: Select your data collection tool(s).
Step 6: Analyze your data.
Turn it into action, measure again
27
Six Steps
to Success
1
2
3
4
5
6
Definitions
Monitoring – process by which data are systematically and regularly collected about a program over time.
Measurement – a way of giving an activity a precise dimension, generally by comparison to some standard; usually done in a quantifiable or numerical manner; see also: data, scale
Metrics – an agreed upon number or percentage that indicates progress towards a goal
Evaluation –a form of research that determines the relative effectiveness of a public relations campaign or program by measuring program*
28
*IPR Dictionary of Measurement Terms, Dr. Donald Stacks www..instituteforpr.org
Why do we communicate?
11/10/2014 29
Outtakes
(Intermediary Effects)
• Awareness
• Knowledge/Education
• Understanding
Outcomes
(Target Audience Action)
• Revenue
• Leads
• Engagement
• Advocacy
Activities How does what you do
contribute to the bottom line?
Step 1: Define the goals
What return is expected? – Define in terms of the
mission.
Define your champagne moment. If you are
celebrating complete 100% success a year from
now, what is different about the organization?
30
Goals & Suggested Metrics
Goals Goal 1: Meet Sales Targets
Goal 2: Reduce Risk/Threats
Goal 3: Increase market share in
new market
Communications Contribution
• Expand the marketable universe
• Reduce sales cycle
• Increase Trust • Increase
advocacy
• Expand the marketable
universe
Metrics
• % Increase in desirable share of voice
• % in awareness
• % increase in undesirable voice
• % increase in trust scores
• % increase in desirable voice in new market
• % increase in awareness of brand in new market
Step 2: Understand the parameters.
What management’s priorities?
Who are you are trying to reach?
How do your efforts connect with
those audiences to achieve the
goal?
What influences their decisions?
What’s important to them?
What makes them act?
32
Goal: Get the cat to stop howling
Options: Local? Cheap? Convenient?
Strategy: Buy cat food
Step 3: Establish benchmarks
Past Performance Over Time
Think 3
Whatever keeps your
C-suite up at night
33
Step 4: Pick your Kick-Butt Index
The Perfect KBI
Is actionable
Is there when you need it
Continuously improves your processes
& gets you where you want to go
You become what you measure, so
pick your KBI carefully
34
The Path to Purchase
Social Media Program
Awareness/Consideration
Preference
Purchase, Retention, Renewal
Kick Butt Quality Score
36
Desirable Criteria Score Undesirable Criteria Score
Contains a key message 3.50
No Key Message -2.0
Contains a desirable visual 1.00
Negative message -3.0
Contains a quote from your spokesperson 2.50
Contains a competitor or opposition quote
-2.0
Positions you as a leader 1.00
A story or a headline that leaves the reader less likely to do business with you
-3.0
The story or headline leaves a reader more likely to do business with you
2.00
Total 10.00
-10.0
Typical Social Engagement Index
Action Score
Like/Follow/Opens/+1/Favorite .5
Share content (including retweets, forwards, Shares etc.) or shares a link to an ENZ site or page. 2
Signs up to receive email or other content or follows a link to content. 2
Contains one or more of the bank’s key messages .5
Positive Comment (leaves a reader more likely to support ) 1
Contains a quote from an alumni or expert 1.5
Views an video 1
Is from a Tier 1 source 1.5
Total 10
37
Step 5: Pick the right measurement tools
If you want to measure messaging, positioning,
themes, sentiment:
Content analysis
If you want to measure awareness, perception,
relationships, preference:
Survey research
If you want to measure engagement, action,
purchase:
Web analytics
If you want predictions and correlations
you need two out of three
All the tools you really need
Google Analytics
Ignore page views
Set up conversions
Excel
11/10/2014
Step 5: Selecting a measurement tool
40
Objective KBI Tool
Increase message
communications
Increase percent of items containing one or
more messages
Media Content Analysis
Increase
awareness/preference
% of audience preferring your brand
to the competition
Survey Monkey, Survata
Engage
marketplace
% increase in engagement on website and/or
social sites
Simply Measured, Google
Analytics, Site Catalyst,
Network Analysis
Step 6: Be Data Informed, not Data Driven
Rank order results from worst to best
Ask “So What?” at least three times
Put your data into an overall framework consistent with C-Suite expectations
Find your “Data Geek”
Compare to last month, last quarter, 13-month average
41 Page 41
Photo Event
High Message Content
Resource Use
Low
Hig
h
Med
ium
Ver
y h
igh
Level of Engagement
Ver
y H
igh
Med
ium
Hig
h
Low
High Resources
No Message
Low Resources
Webinar
Status update
Link
Ultimate Road Trip
Google + Chat
Media Day
Corporate Video
Advocacy vs. resource use
Resource Use
Low
Hig
h
Me
diu
m
Ve
ry h
igh
Ver
y H
igh
Med
ium
Hig
h
Low
High Resources Low Resources
Webinar
Status update
Link
Ultimate Road Trip
Google + Chat
Media Day
Corporate Video
Real-Time Metrics focus on Outcomes
43
PUBLIC AFFAIRS & BRAND DASHBOARD Priority 1: Brand Awareness Priority 2: Advocacy Priority 3: New Membership Culture
Bench-mark
Current Variance Bench-mark
Current Variance Bench-mark
Current Variance Bench-mark
Current Variance
% increase in share of desirable voice
8.9% 2.17% ↓6.73 pts
% increase in PRINZ share of Desirable Conversation around Digital privacy
10% 15 ↑6.73 pts
% Increase in lawyers who are extremely or very familiar with PRINZ’s positioning
38% 38% Annual Data Only
% increase in ratings on communications trustworthiness (from culture survey)
54% New data
available July 14
N/A
% decrease in share of undesirable voice
1% 1% Un-changed
% decrease in Government officials unaware of PRINZs position on Digital Privacy
10% 10% ↑6.73 pts
% Increase in online engagement on issue
25% 0 1% % increase in employee knowledge of PRINZ “Way We Work” (From Knowledge Quizzes)
23% New data
available Q3
N/A
% increase in awareness among lawyers (from 2014 Brand Metrics Survey)
46% 46% Next data
available Q3 14
% increase in Inbound info requests for info
516(Apr) 427 (May)
↓17%
% employees strongly understanding strategic objectives (From Knowledge Quizzes)
32% New data
available Q3
N/A
% increase in perception of PRINZ as a leader in Digital Privacy
5% 5% Un-changed
% increase in engagement on PRINZ.org.nz
202K
197K
↓2%s
% reduction in Say/Do gap on agility metrics (average of agile & make decisions faster)
36.5% New data
available Q4
N/A
Below Target
At Risk
On Target
Connecting data yield greater insights
106 38 24
73 101 49 25
1087
1331
699 637
848
1331
699
10.25
35.03
29.13
8.73 8.40
27.16 27.96
0
5
10
15
20
25
30
35
40
0
200
400
600
800
1000
1200
1400
Jan Feb Mar April May June July
Relationship between interviews and placements
Number of Interviews Number of placements Interview per placement
ACA has made a demonstrable difference in the quality of
media coverage
If an ACA or one of its
programs was mentioned
in a media story, it was:
More likely to contain
an endorsement
More likely to be
positive
More likely to contain
key messages
Less likely to be
negative
0.22
0.00
0.62
1.42
4.61
0.08
0.08
0.01
-0.61
-0.66
Endorsements
Positive Visual
Negative Visual
Tone
MessageCommunication
Average of OCS
Differences in Quality of Coverage with and without
ACA
Atlantic City Atlantic City Alliance
Without ACA events, OCS Scores for Atlantic City would
have been significantly lower
4
3.25
2.75 2.99
3.65
2.96 3.36 3.24
2.34 2.37 2.43
1.30
-1.24
0.37 -0.05
0.28 0.28
-1.56
4.91
3.92
2.99
3.58
4.14 4.1 4.27 4.12 4.29
2.78 2.56
1.53
-0.29
0.61 0.20
1.44 1.77
0.63
-2
-1
0
1
2
3
4
5
6
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June
Ave
rage
OC
S Sc
ore
Average OCS Score Over Time
Atlantic City OCS without ACA Atlantic City OCS with ACA
The red line represents coverage of Atlantic City minus all mentions of ACA and its programs
Free entertainment generated the highest OCS Scores in
Q2
5.26
5.73
6.36
6.54
6.71
7.15
7.31
7.36
7.50
7.67
8.00
8.10
8.68
9.30
Miss America
Meet AC
DO AC
July 4th Fireworks
Air Show
Blake Shelton
Miss'd America Pageant
Sand Blast
Hello Summer
Boardwalk Hall Light Show
Challenge Triathlon
Sand Sculpting World Cup
Lady Antebellum Concert
Free Entertainment
Top Programs By OCS
When ACA programs received media coverage, traffic
followed
0
50
100
150
200
250
300
0
20000
40000
60000
80000
100000
120000
140000
Relationship between ACA Program Mentions and Site Visits
Site Visits Program Mentions
ACA programs drive higher OCS scores, which correlate
highly with web visits
0.41
0.44
0.47
AC items
ACA Items
ACA OCS Scores
Correlations between Web Visits and PR Metrics
Pearson r. value
White House Volunteerism Office (CNCS) is able to connect
specific social outreach to registrations on serve.gov
51
The ROI of Emily – So What?
52
Red line
indicates
media
impressions
35,152,789 OTS
So What = Revenue
53
10 Ways to find the Money for Measurement
1. Don’t call it measurement – It’s Research
2. Consolidate - Social? IR? Agencies? Research?
3. Crowdsource – Who else has tools or surveys?
4. Spread the cost over 2 fiscal years
5. Reach out to universities
6. Take advantage of free stuff
7. Monitor only what matters
8. Narrow your Top Tier media list to what you CAN afford
9. Google analytics
10. Facebook Insights
11. Twitter Analytics
Remember These Points
It’s not about the media, it’s about the business and the customer 1
It’s not about Big Data, but about how you use it. 2
You need to be data informed, not data-driven 3
It’s not how loud you’re shouting it’s about relationships. 4
Standards are a reality not an excuse to hide behind 5
Thank You!
For more information on measurement, read my blog:
http://www.painepublishing.com/blog
For a copy of this presentation or to subscribe to our newsletter, give me
your card or email me at [email protected]
Follow me on Twitter: @queenofmetrics
Friend me on Facebook: Katie Paine
Or call me at 1-603-682-0735
56