Quantifying the "Buzz" Effect: Integrating Social Media with Loyalty

& Defection ModelsMarketing Associates:

Keith Shields, Director, Decision SciencesRoni Leibovitch, Senior Consultant, Digital Intelligence

Mindy Deatrick, Senior Consultant, Quantitative SolutionsFord Motor Company:

Margaret Kishore, Performance and Metrics Manager

About the Title…

“Buzz” refers to the amount of, and sentiment of, the Ford-related comments available through social media outlets.

The “Buzz Effect” refers to the increase or decrease in brand loyalty / defection (measured by repurchase) that occurs as a result of a change in the Buzz.

“Quantifying the Buzz Effect” means we want to put a number on the amount of that increase or decrease.

The advantage of this is that we can begin to put a dollar value on salient, publicly-known events…such as refusing to take government bailouts.

“Integrating Social Media With Loyalty / Defection Models” means that we will: Extract signals of future vehicle purchase decisions from customer comments found

through social media outlets AND Capture those signals in the form of predictive variables to put into loyalty models. Those variables and their associated model coefficients will quantify the buzz effect. For the purpose of this analysis we focus our efforts on Twitter.

1

Warnings and Disclaimers

We will do our best to reveal trends, patterns, and findings without showing actual numbers (but for some cases). Hiding / changing of numbers is done to protect the innocent (Ford Motor Company especially).

In the course of the presentation we will share many Ford-related “tweets”. These will be actual tweets. They will not be censored because their informal nature highlights a point we want to make about text mining. Please try not to be offended.

We will not be presenting any astounding research and/or revelations about text mining. We use “off-the-shelf” techniques when it comes to categorizing sentiment.

Our expertise is in modeling and predicting customer behavior based on all available and relevant customer data.

We see social media as a potentially rich source of customer data, and those data just happen to be free-form text.

2

One More Item of Note…

3

Background on Ford’s Social Media Efforts…

Measuring the “Consumer Experience” Alan Mulally and Apple… The Dealership Experience: Sales and Service The Ownership Experience How do people share experiences? Traditionally by talking to each other. But

how much today is done through Twitter, Facebook, Blogs?

By analyzing the comments and sentiment expressed through Social Media outlets can we glean meaningful insights about the Ford Consumer Experience?

Can we make inference about a consumer’s affinity for Ford…or an existing customer’s loyalty to Ford?

If no, then we’re probably not trying hard enough. Examples next 2 slides.

4

Google Twitter Search: Ford Comments

Search: “My Ford Focus is great.”

I love my Ford Focus, but not so much Ford Service in Northampton Mass. Thieves.

Got my new computer yesterday and can't wait to get my new 2012 Ford Focus SEL in 4-6 weeks! 23 Apr

Am test driving Hondas and Fords 7 Apr

We’d like to have a mechanism for intervening here. On April 7 this person indicated he was facing a choice between buying a Honda and buying a Ford.

Does this mean we can simply scrape Twitter for the words “test drive”? Seems like it would be predictive of future behavior…

5

Google Twitter Search: Ford Comments

Search: “I don’t like my Ford Escort.”

The ford escort texting and driving, I really like my life and my car, please don't try and drive into us, twice. Close call!

My old '93 ford escort is running 130k and runs like a charm.... And my 2003 ford ranger truck has 80k without problems.

Again this seems like something that, if captured and quantified in the form of a variable, would be predictive in the context of a loyalty model.

6

Google Twitter Search: Ford Comments

Search: “Ford, government bailout” This weekend my wife and I purchased a FORD. Why? Because they

chose not to accept the government funded bailout. Ford didn't accept the government bailout - that's pretty awesome. Wait #Ford pulled the ad that was critical of the #Obama bailout but is

now running one that jokes about drinking and driving? GM CEO wants higher gas tax. Buy a Ford car or truck. Please RT

This is an example of how capturing “influencers” could be very important. This person happens to have 340 followers and routinely tweets about auto-related topics.

So the effort to mine Twitter for Ford sentiment extends beyond improving the loyalty and defection models…but the title of this presentation does not. That said, we will discuss how we are affecting marketing programs with our existing knowledge of influencers.

7

Background on Ford’s Social Media Efforts…

Measuring the “Consumer Experience” Alan Mulally and Apple… The Dealership Experience: Sales and Service The Ownership Experience How do people share experiences? Traditionally by talking to each

other. But how much today is done through Twitter, Facebook, Blogs?

By analyzing the comments and sentiment expressed through Social Media outlets can we glean meaningful insights about the Ford Consumer Experience?

Can we make inference about a consumer’s affinity for Ford…or an existing customer’s loyalty to Ford?

Yes! So what can we do capitalize upon good sentiment and reverse bad sentiment?

8

Start With the Current Infrastructure The Ford Motor Company has a customer data warehouse that collects relevant data from all

customer touchpoints, “customerizes” it, and applies a suite of predictive models that are used for targeted campaigns.

More importantly the warehouse is connected to many customer-facing and dealer-facing operational systems, and it passes important information about customer behavior, both past behavior and predicted behavior, to operational systems when decisions regarding the customer have to be made in real time.

Data Refinery “Customerization”

LOYALTY AND DEFECTION MODELS APPLIED

Call Center

Website

Dealers Click-thru Mail Call Center

Sales Dealer

Users & business interface

Data Creation

Customers

Customer Data by Touch Point

User-facing systems easily access scores and strategies in real time.

ETLCustomer Data Warehouse

Service

Survey

9

Fitting in to the Current Infrastructure…

Social media is just another customer touch point. The text we mine from social media outlets is another set of data about the customer,

just like the call center, website, or the customer surveys. We’d like to use that data just like the rest of the customer data: to help us predict customer

purchase behavior. In some sense social media provides a source of unsolicited surveys.

Data Refinery “Customerization”

Call Center

Website

Dealers Click-thru Mail Call Center

Sales Dealer

Users & business interface

Data Creation

Customers

Customer Data by Touch Point

User-facing systems easily access scores and strategies in real time.

ETLCustomer Data Warehouse

Service

Survey Comments / Sentiment

LOYALTY AND DEFECTION MODELS APPLIED

Twitter, FB, …

10

A Source of “Unsolicited Surveys”…

Why do we survey customers? From the narrow perspective of someone who predicts customer behavior, the graph below is a big reason why.

How much do we spend on surveys? Whatever it is, our feeling is that if we can establish the above relationship with social media

sentiment (use it as your X-axis), and cover more customers for less than what we currently spend on surveys, then we have the beginning of a business case for extracting sentiment from social media.

What’s more compelling is that relationship between a customer’s opinion and loyalty holds up when we control for predicted loyalty.

The “Loyalty=1” group is the group that scores in the lowest 20% of a loyalty model…a model built without survey data.

The results remain consistent within each loyalty tranche so much so that customers within group 5 can have lower repurchase rates than those in group 3, depending on survey response.

11

We believe that there are three “pillars” for the business case to actively engage consumers through social media outlets (specifically Twitter):1. Conquesting new customers2. Concern resolution3. Voice of Customer

Introduce Social Media as an additional consumer touch-point

1. Conquest$XX mils per year

STRATEGY:

SUPPORTS THESTRATEGY:

2. Concern Res$XX mils per year

3. VOC$XX mils per year

A Quick Digression on Business Cases…

12

How We Use Survey Data In the Models…

Let P = probability a Ford customer will repurchase another Ford upon disposing of any one of his current Fords.

Logistic regression is a very popular way to model and predict P. ln[p / (1-p)] = b0 + b1*x1 + b2*x2 + … bn*xn

b0, b1, b2 are parameter estimates. They quantify the extent to which x1, x2, …, xn affect the probability of repurchase.

x1, x2, …, xn are explanatory variables, e.g. # of previous Ford purchase, time since most recent Ford purchase, miles from nearest Ford dealership, etc…

Now let s1 = 1 if “very likely”, 0 otherwise Let s2 = 1 if “likely”, 0 otherwise … Let s5 = 1 if “not at all very unlikely”, 0 otherwise Refit the logistic regression:

ln[p / (1-p)] = b0 + b1*x1 + … bn*xn + bn+1*s1 + bn+2*s2 + bn+3*s3 + bn+4*s4 + bn+5*s5

Can be thought of as the VOC (Voice of Customer) Index, but it’s based on just survey data, which may only be available on 10% (roughly) of the customers.This is a nice metric, because it, by design, predicts loyalty.

13

How We Use Twitter Data In the Models…

Let’s treat the Ford customer’s “tweets” the same way we treat survey data. Go back to our logistic regression:

ln[p / (1-p)] = b0 + b1*x1 + … bn*xn + bn+1*s1 + bn+2*s2 + bn+3*s3 + bn+4*s4 + bn+5*s5

And let t1 = 1 if we can identify a “Ford positive” tweet for the customer, 0 otherwise. Let t2 = 1 if we can identify a “Ford neutral” tweet for the customer, 0 otherwise. Let t3 = 1 if we can identify a “Ford negative” tweet for the customer, 0 otherwise. Refit the model:

ln[p / (1-p)] = b0 + b1*x1 + … bn*xn + bn+1*s1 + bn+2*s2 + bn+3*s3 + bn+4*s4 + bn+5*s5

+ bn+6*t1 + bn+7*t2 + bn+8*t3

Can be thought of as the BUZZ INDEX, and it comes directly from what ford customers are saying on Twitter. This metric also, by design, predicts loyalty. So this is a quantification of the BUZZ EFFECT.

14

Interpreting the “Twitter-Enhanced” Model…

ln[p / (1-p)] = b0 + b1*x1 + … bn*xn + bn+1*s1 + bn+2*s2 + bn+3*s3 + bn+4*s4 + bn+5*s5

+ bn+6*t1 + bn+7*t2 + bn+8*t3

When we fit this model, we get an intuitive result: bn+6 > bn+7 > bn+8 => good tweets lead to higher loyalty than do neutral tweets,

neutral tweets lead to higher loyalty than bad tweets.

Not as intuitive (but interesting nonetheless): All three parameters are greater than 0 (implying ANY tweeting is better than

no tweeting). bn+8 (the parameter for bad tweets) is NOT SIGNIFICANT. There is not a

sufficient volume of bad tweets to support a significant result. The large majority of FLM tweets are good.

We think that the upshot of all of this is that tweeting about Ford, irrespective of sentiment, signifies a high level of customer engagement. This has implications beyond our efforts to better predict loyalty.

15

The Buzz Variables Improve the Loyalty Model: So What?

More data and better data yield models that do a better job of “separating” loyalists and non-loyalists.

One way this manifests itself: ranking the population of customers with a better model will yield higher repurchase rates in the top decile (or demi-deciles…depending on how many groups you want to establish), and lower repurchase rates in the bottom decile.

So a marketing campaign that increases everyone’s likelihood of repurchase by 15% (not an uncommon number), does so on a larger base of loyalists within the top decile, and thus creates more incremental sales for the same amount of mailings.

16

1 2 3 4 5 6 7 8 9 10

Old ModelModel w/VOC & Buzz

Model DecileLow Loyalty to High

Repu

rcha

se R

ate

Say the difference between these two bars is 200 bps. Some of the incremental sales from the campaign

noted in the bullet above (top decile only), are attributable to having a better model.

How many? .02 * .15 * top decile population If the population of interest is 250,000 customers, then

the impact of the better model is 750 incremental repurchases.

If a repurchase is worth $5,000 profit, then the case for the “buzz variables” is substantial: $3.75 million.

Why We Will Regularly Re-Fit the Buzz Index

We have several reasons to believe that results may change when we re-fit our enhanced loyalty model (twitter sentiment data being the enhancement):

1. The number of Tweeters is increasing all the time. Ford’s customer email capture isn’t great but it is improving, and there is evidence that Ford customers are, relatively speaking, very active on Twitter.

2. Attribution of tweets to customers is difficult and unsure; finding the Twitter names of Ford customers is difficult and painstaking.

3. Classifying the sentiment of Tweets is an imprecise exercise, especially when using off-the-shelf tools and software.

4. The content of the “Ford-tweeting” population leads to potentially biased results; it is biased toward a demographic that naturally tends to be less Ford loyal:

Females Young Used vehicle owners Appear to be more service loyal, which is a good thing

5. The tweeting population also happens to be geographically biased, but this does not concern us as much as #4.

17

Re-Fit the Buzz Index: The Increasing Number of Tweeters

18

According to GIGAOM (http://gigaom.com/), Twitter had 175 million users in December 2010, and was growing by 370,000 new users every day. Also as of 12/2010:

65% of those users lived outside the US. Roughly 6% of all Americans were active on Twitter. More recent studies indicate the number is 9%-10% (or

13% of internet users). Of the Ford customers active as of 12/2010 (who had a valid email), we were able to find 9% of

them active on Twitter. Not surprisingly we have found the number of Ford related tweets to be increasing over time.

Good and neutral tweets have increased whereas bad tweets have stayed flat. Seems like a good thing, but we have some thoughts on comment classification.

Frequency of Ford-Related Comments Found on Twitter

0

10,000

20,000

30,000

40,000

50,000

60,000

May

-08

Jul-0

8

Sep-

08

Nov

-08

Jan-

09

Mar

-09

May

-09

Jul-0

9

Sep-

09

Nov

-09

Jan-

10

Mar

-10

May

-10

Jul-1

0

Sep-

10

Nov

-10

Jan-

11

Mar

-11

Ford-Related Twitter Comments "Binned"

0

1000

2000

3000

4000

5000

6000

May

-08

Jun-

08Ju

l-08

Aug-

08Se

p-08

Oct

-08

Nov-

08De

c-08

Jan-

09Fe

b-09

Mar

-09

Apr-

09M

ay-0

9Ju

n-09

Jul-0

9Au

g-09

Sep-

09O

ct-0

9No

v-09

Dec-

09Ja

n-10

Feb-

10M

ar-1

0Ap

r-10

May

-10

Jun-

10Ju

l-10

Aug-

10Se

p-10

Oct

-10

Nov-

10De

c-10

Jan-

11Fe

b-11

Mar

-11

Posti

ve, N

egati

ve

0

10000

20000

30000

40000

50000

60000

Neu

tral

Negative

Positive

Neutral

Re-Fit the Buzz Index: Attributing the Tweets

19

Twitter names can be found if you can supply an email.

With an email address you can find, through the Twitter API, a Twitter name, a first name, and a last name associated with that email address. It will not return the email.

Ford has 11 million bought-new still-retained customers. We have emails on several million of them.

We cannot run several million emails through the Twitter API. Even if they could be processed, we would not be able to get back the email. We would only get back the thousands of Twitter names associated with, but not matched to the emails.

The only sure way to attribute emails to Twitter names is to go through the API one email at a time. Can we off-shore this? We can, but we still will not be done until 2013 if we go this route. So we had to be more clever…

Whatever method we choose, we need to recognize the Twitter name as useful customer data, and as such, store it in the data warehouse. Next slide…

Re-Fit the Buzz Index: Attributing the Tweets – You Must Retain Data

20

What comes out of the attribution process is a table that looks like this:

We have another process (using RADIAN6) that scours Twitter for comments that contain words in our “start list” (e.g. Ford, Lincoln, Mercury, Taurus, Mustang, Fusion, etc…). It produces a table that looks like this:

EMAIL TWITTER_NAME FIRST_NAME LAST_NAMEkshields@marketingassociates.com Keith Shields

rleibovitch@marketingassociates.com ronedog Roni Leibovitchmdeatrick@marketingassociates.com Mindy Deatrick

TWITTER_NAME COMMENT SENTIMENTronedog My new Ford Focus is also imported from Detroit. Good

Re-Fit the Buzz Index: Attributing the Tweets – You Must Retain Data

21

Data Creation

Customer Data Warehouse

Twitter

We have to integrate this into existing data warehouse processes (which should be easy enough, if we’re treating this like just another source of customer data):

EMAIL TWITTER_NAME FIRST_NAME LAST_NAMEkshields@marketingassociates.com Keith Shields

rleibovitch@marketingassociates.com ronedog Roni Leibovitchmdeatrick@marketingassociates.com Mindy Deatrick

Customer Touchpoint

Data RetentionTWITTER_NAME COMMENT SENTIMENT

ronedog My new Ford Focus is also imported from Detroit. Good

Re-Fit the Buzz Index: Classifying the Tweets…

Prepackaged comment binning algorithms are not as accurate as we’d like…they result in a high instance of inappropriate comment binning (“Positive”, “Neutral”, or “Negative”). Here are some actual examples of inappropriately binned comments:

Positive: “Classic Car For Sale 2001 FORD EXPLORER - Mt. Royal NJ: Runs and Looks Great!!!” Negative: “You have insulted my Ford Fiesta, shame on you.” AND “Just drove a Ford Fiesta

getting 30 mph. Not bad!” AND “Just dropped my car off at the FORD Dealership . I want a FORD Fusion soooooo BAD.”

Neutral: “Ford Escape!!!” AND “Smart, easy, & fun ride with the new Ford Focus with Ford Sync's help!”

In order for us to get the intuitive results we showed on slide 12 we had to depart from sentiment classification algorithms and do a “brute-force” classification.

Interns, Cornerstone Schools, Detroit, Mi. (http://www.cornerstoneschools.org/) There are other inexpensive ways…all of which we believe to be more accurate than existing

“machine intelligence”…albeit not as scalable:• Existing call center personnel• Mechanical Turks (https://www.mturk.com/mturk/welcome)

22

Re-Fit the Buzz Index: The Biased, But Changing, Population of Tweeters…

See the graph on the right. The fastest growing population of tweeters is 18-34 year olds. About 56% of tweeters are 34 years or younger.

23

The challenge for Ford: the median age for Ford customers is well above 34, despite some recent strong entries in the small car market.

The most “tweeted about” Ford is, not surprisingly, the Focus.

Re-Fit the Buzz Index: Geographically-Biased Population of Tweeters…

The numbers represent how much higher the Twitter use per capita is in that state versus the nation as a whole.

For example: if the national usage rate is 10%, then Michigan is 11% lower than that: .10 - .11(.10) = 8.9%.

24

Given the Worries About Attribution, Classification…

25

The real opportunity could lie in the business experts using intuition and common sense to tailor campaigns and programs to tweeters based on their most recent comments.

This would only rely upon a good mechanism for scraping relative comments from Twitter and reacting procedurally and appropriately.

If we look at the comments as unsolicited survey responses we see opportunities for customized offers and programs (no models needed – the comment reveals the customer’s intent):

Private Sales Offer and/or Pre-approval: “Just dropped my car off at the FORD Dealership. I want a FORD Fusion soooooo BAD.”

Rewards Program Offer: “My moms taking me to get this Ford explorer in the mornin tho i should have a new whip before July then im haulin ass to the A”

Offer for trade-in: “Ford focus sucks. Very uncomfortable vehicle.” Offer for service discount / extended warranty: “My car is running rough and

keeps blowing the injector and on plug coil fuses, its a 2006 3.0 V6 Ford Fusion. HELP!!”

We believe that there are three “pillars” for the business case to actively engage consumers through social media outlets (specifically Twitter):1. Conquesting new customers: pay attention to influencers2. Concern resolution3. Voice of Customer

Introduce Social Media as an additional consumer touch-point

1. Conquest$XX mils per year

STRATEGY:

SUPPORTS THESTRATEGY:

2. Concern Res$XX mils per year

3. VOC$XX mils per year

So We Revisit Our Three Pillars…

26

Conquest New Customers: Influencers

27

Fact: 93.6% of Twitter users have less than 100 followers, while 98% of users have less than 400 followers. Meanwhile, 1.35% of users have more 500 followers, and only 0.68% of more than 1,000 followers.

Conquest New Customers: Influencers

28

Fact: As Twitter users attract more followers, they tend to Tweet more often. This is particularly evident once someone has 1,000 followers the average number of Tweets/day climb from three to six. When someone has more than 1,750 followers, the number of Tweets/day rises to 10.

Fact: A small group of Twitter users account for the bulk of activity. Sysomos discovered that 5% of users account for 75% of all activity, 10% account for 86% of activity, and the top 30 account for 97.4% of activity.

Conquest New Customers: The Opportunity

Find out who is expressing in-market sentiment and send them a targeted offer.

We estimate that through Twitter alone, roughly 35,000 customers per year express inclination to buy Ford.

Applying a result from an analysis of "handraiser campaigns", we assume 15% of the 35,000 will purchase FLM. This is 35,000 * 15% = 5,250 sales.

Assuming 20% lift from a targeted offer to in-market customers (not an uncommon number), we estimate that a conquesting campaign directed at in-market "social-media leads“. This is 5,250 * .2 = 1,050 incremental sales.

In 1Q2012 Ford will run a program that tests the opportunity we have framed in the above bullets. We will have the capability to offer differentiated treatment to influencers. Marketing Associates built the process to find the tweets and measure the back-end results.

INTEGRATION will be through the customer data warehouse and EXECUTION through the concern resolution center. At this point we won’t trouble you with another infrastructure diagram.

29

Some recommendations…

Treat social media as another source of relevant customer data. Comments about your product are, in some sense, unsolicited surveys. They can, like surveys do,

improve your ability to predict the behavior of your own customers.

Pay careful attention to the integration of social media data. Integration requires “customerization”, so subsequent customer behavior can be tracked. Attributing comments to customers is tricky. It can also be painstaking. The good news is that it

can be done cheaply.

Correct classification of comments is essential to understanding the true signal in the comments. The most accurate means of classification may also be the least scientific: have an English

speaker (who preferable understands colloquialisms) read the comments, and bin them. Categories can be “good”, “bad”, “in-market”, “service issue”, or whatever aligns with the

differentiated treatments and offers.

Retain data, and integrate intelligently into a data warehouse. Test and measure several tactical approaches to customers and prospects who are commenting about your products.

30

30

Quantifying the "Buzz" Effect: Integrating Social Media with Loyalty & Defection Models

Questions?

Thanks for your time and attention.

30

APPENDIX From www.dealer.com:

58% of vehicle purchasers posted a comment or status update on Facebook about their new vehicle.

41% of those who used social media while shopping, said they saw a post that caused them to add a brand or model to their consideration. 28% saw a posting that caused them to add a dealership to their consideration.

74% of those who use Facebook said they would likely "Like" a manufacturer if they were offered exclusive offers and savings.

Methodology:Approximately 2,000 recent vehicle purchasers and in-market shoppers were surveyed about their use of social media. The study was based on McKinsey Consulting's consumer decision journey model. The research was conducted by GfK Automotive.