Using AI to Make Sense of Customer Feedback
-
Upload
alyona-medelyan -
Category
Data & Analytics
-
view
403 -
download
2
Transcript of Using AI to Make Sense of Customer Feedback
![Page 1: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/1.jpg)
Using AI to Make Sense of
Customer Feedback
Alyona Medelyan
@zelandiya
![Page 2: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/2.jpg)
Correct Understanding of Customer Feedback
Can Save Millions
![Page 3: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/3.jpg)
2015: Tens of Thousands of New Zealanders
were Surveyed About the new Flag
![Page 4: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/4.jpg)
Government Reported
the Results of Manual Feedback Analysis
![Page 5: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/5.jpg)
Actual Responses
Two costly & unnecessary referendum followed. Outcome: NZ kept the current flag
Millions could have been saved!
People wanted to ”keep the current flag”
![Page 6: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/6.jpg)
1. Types of customer feedback
2. Why analyzing customer feedback is important
3. Why is it hard
4. Approaches
5. Applying AI to customer feedback analysis
6. Demo
![Page 7: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/7.jpg)
Different Types
of Customer Feedback
![Page 8: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/8.jpg)
Types of Customer Feedback
one-on-one interviews / focus groups
call centre logs / complaints
social media
open-ended survey questions / reviews
quantitate survey questions
UX tests / analytics
unstructured
structured
![Page 9: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/9.jpg)
Collection Analysis Insight
one-on-one interviews / focus groups hard hard good
call centre logs / complaints easy hard limited
social media easy hard limited
open-ended survey questions / reviews easy medium good
quantitate survey questions easy easy limited
UX tests / analytics medium easy limited
unstructured
structured
Comparing Types of Customer Feedback
![Page 10: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/10.jpg)
Why Understanding
Customer Feedback
is More Important than Ever
![Page 11: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/11.jpg)
Customer Experience
is the New Marketing
It’s Measured Using
Net Promoter Score Surveys
Image credit
![Page 12: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/12.jpg)
![Page 13: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/13.jpg)
The number of “Net Promoter Score”
searches on Google since 2004
1. Growing Number of
Satisfaction Surveys and Reviews
![Page 14: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/14.jpg)
v
¯\_(ツ)_/¯
2. The Need to Explain
the Why’s Behind the Scores
Net Promoter Score by month over time
![Page 15: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/15.jpg)
3. Scores can be Cheated
Unstructured Feedback, not so Much
![Page 16: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/16.jpg)
Why Analyzing
Customer Feedback is Hard
![Page 17: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/17.jpg)
Common Misconception:
Sarcasm Makes Analysis Hard
One of Many Sarcastic Tui Beer Adverts
![Page 18: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/18.jpg)
Sarcasm is Hard: Even People Struggle
I’ll keep it in
mind
They’ll do itI’ve
forgotten
already
![Page 19: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/19.jpg)
Sarcasm is Rarer Than You Think
Dataset Sarcasm Example
NPS Survey 1%I’m so disappointed! What a great
customer service you have!
Social Media
comments5% Very helpful answer. Troll.
![Page 20: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/20.jpg)
The Actual Challenges
With Customer Feedback
![Page 21: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/21.jpg)
Challenge 1: Messy Data
![Page 22: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/22.jpg)
How many ways there are to say
‘wet paper’?
Challenge 2: Synonyms and Paraphrases
![Page 23: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/23.jpg)
Hundreds of
possible variations
of the same theme
wet
dripping
soaking
soaked
damp
drenched
paper
papers
newspaper
news paper
newspapers
news papers
+
Paraphrasing the Same Theme
![Page 24: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/24.jpg)
Challenge 3: Negation
Positive or Negative?
My coffee was great positive
My coffee was awful negative
My coffee was not great negative
My coffee was not that great neutral?
I did not think my coffee was great negative
I did not expect my coffee to be this great positive
I was disappointed with the quality of the coffee negative
I was not disappointed with the quality of the coffee positive
![Page 25: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/25.jpg)
Approaches to
Customer Feedback Analysis
![Page 26: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/26.jpg)
Manual Coding
1.
![Page 27: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/27.jpg)
Figure out the Code Frame, Apply, Repeat
What is the meaning of life?
1 2 3 4 5
What is the meaning of life?
42
Friends and family
Making a difference in the world
Happiness
Finding happiness
To achieve, to conquer
Family
…
What is the meaning of life?
42
Friends and family
Making a difference in the world
Happiness
Finding happiness
To achieve, to conquer
Family
…
1
2
3
4
4
5
2
![Page 28: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/28.jpg)
Sentiment in a Manual Code Frame
Customer Service
Positive Negative
Timely Nice Helpful Didn’t fix issue Rude
![Page 29: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/29.jpg)
Word Clouds
2.
“Every time I see a word cloud presented as insight,
I die a little inside.”
– J. Harris, journalist
![Page 30: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/30.jpg)
Word Clouds Lack
Interpretation, Context, Meaning
“Overall the language
focuses on sweeping
statements focusing on
the state of the nation.”
Kalev Leetaru (Forbes)
![Page 31: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/31.jpg)
You wouldn’t create a Word Cloud from your Numbers,
why is it ok from Text?
![Page 32: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/32.jpg)
Rule-based Approaches
3.
![Page 33: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/33.jpg)
It’s Hard to Find a Rule That Works Well
I was impressed by how friendly the person
on the other end of the line wasStaff friendliness ✔
The lady who helped me was friendly Staff friendliness ✔
Friendliness of staff Staff friendliness ✔
Your website is very user friendly Staff friendliness ✘
The young man on the phone was very pleasant Other ✘
friendly OR friendliness –> Staff friendliness
![Page 34: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/34.jpg)
Text Categorization
4.
![Page 35: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/35.jpg)
old
customer
responses
categories
new
customer
responses
Machine
Learning
Algorithm
Predictive
Model categories
Need for Sufficient Training Data,
and Clear Categories
![Page 36: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/36.jpg)
Customer Feedback Analysis
Needs to be ‘Unsupervised’
![Page 37: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/37.jpg)
Thanks to an unsupervised approach, Facebook found
Candi Crash Saga causes low App Store reviews
![Page 38: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/38.jpg)
Topic Modeling
5.
![Page 39: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/39.jpg)
21
3
A Topic can be Hard to Interpret
2
???ok
Source: Ben Fields
![Page 40: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/40.jpg)
Sentiment
![Page 41: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/41.jpg)
1. Rule-based (dictionary)
2. Text categorization (positive / negative)
Two Sentiment Detection Approaches
![Page 42: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/42.jpg)
Advances in AI > Customer Feedback
![Page 43: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/43.jpg)
Messy Data
Paraphrases
Negation
AI > Challenges
Word2vec*
Deep Learning
*See also: Conceptnet.io
Knowledge Representation
![Page 44: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/44.jpg)
Word2Vec
Image source: ericbern.com
Best Intro: Word2Vec Udacity Youtube
![Page 45: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/45.jpg)
Knowledge Representation
![Page 46: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/46.jpg)
Deep Learning
Precision Recall F-Measure Errors
People 84 73 75 <1
Dictionaries 61 57 54 8
Linear Regression 65 56 47 3
Deep Learning 62 57 49 2
Sentiment Analysis is not about maximizing F-Measure,
it’s about reducing true Errors: positive confused with negative
![Page 47: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/47.jpg)
Theme Extraction
6.
![Page 48: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/48.jpg)
From Words to Complex Themes
![Page 49: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/49.jpg)
Applying Customer Feedback Analysis
![Page 50: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/50.jpg)
Google: Sentiment by Theme
![Page 51: Using AI to Make Sense of Customer Feedback](https://reader034.fdocuments.net/reader034/viewer/2022051710/5a67b7787f8b9a360c8b6ecb/html5/thumbnails/51.jpg)
Thematic Demo