Interaction Mining: the new frontier of Call Center Analytics

Interac(on Mining: the new fron(er of Call Center Analy(cs

Vincenzo Pallo:a Rodolfo Delmonte Lammert Vrieling David Walker

© 2011 interAnaly(cs 1

Outline

•  Call Center Analy(cs •  Automa(c Argumenta(ve Analysis for Interac(on Mining

•  Experiments with Call Center Data •  Conclusions


CALL CENTER ANALYTICS


Call Center Analy(cs

•  Call centers data represent a valuable asset for companies, but it is oOen underexploited for business purposes because: –  it is highly dependent on quality of speech recogni(on technology

–  it is mostly based on text-‐based content analysis. •  Interac(on Mining as a viable alterna(ve: –  more robust –  tailored for the conversa(onal domain –  slanted towards pragma&c and discourse analysis


Mainstream Call Center Analy(cs


Does not u

nveil real

insights abo

ut custom

er

sa=sfac=on

Call Center Analy(cs: metrics and KPIs

•  Agent Performance Sta(s(cs: –  Average Speed of Answer, Average Hold Time, Call Abandonment Rate,

A<ained Service Level, and Average Talk Time. –  Quan(ta(ve measurements that can be obtained directly through ACD

(Automa(c Call Distribu(on), Switch Output and Network Usage Data.

•  Peripheral Performance Data: –  Cost Per Call, First-‐Call Resolu&on Rate, Customer Sa,sfac,on, Account

Reten&on, Staff Turnover, Actual vs. Budgeted Costs, and Employee Loyalty. –  Quan(ta(ve, with the excep(on of Customer Sa&sfac&on that is usually

obtained through Customer Surveys. •  Performance Observa(on:

–  Call Quality, Accuracy and Efficiency, Adherence to Script, Communica,on E,que;e, and Corporate Image Exemplifica,on.

–  Qualita=ve metrics based on analysis of recorded calls and session monitoring by a supervisor.


Four objec(ves

1.  Iden(fy Customer Oriented Behaviors, –  which are highly correlated to posi(ve customer ra(ngs (Rafaeli et al. 2007);

2.  Iden(fy Root Cause of Problems –  by looking at controversial topics and how agents are able to deal with them;

3.  Iden(fy customers who need par(cular a:en(on –  based on history of problema(c interac(ons;

4.  Learn best prac(ces in dealing with customers –  by iden(fying agents able to carry coopera(ve conversa(ons.


ARGUMENTATIVE ANALYSIS FOR INTERACTION MINING


Argumenta(ve Structure of Conversa(ons

DISCUSS(issue) <- PROPOSE(alternative) 1702.95 David: so - so my question is should we go ahead and get na- - nine identical head mounted crown mikes ? {qy} 61a

REJECT(alternative) 1708.89 John: not before having one come here and have some people try it out . {s^arp^co} 61b.62a

PROVIDE(justification) 1714.09 B: because there's no point in doing that if it's not going to be any better . {s} 61b+

ACCEPT(justification) 1712.69 David: okay . {s^bk} 62b

PROPOSE(alternative) 1716.85 John: so why don't we get one of these with the crown with a different headset ? {qw^cs} 63a

ACCEPT(alternative) 1721.56 David: yeah . {s^bk} 63b 1726.05 Lucy: yeah . {b} 1727.34 John: yeah . {b}

PROVIDE(justification) 1722.4 John: and - and see if that works . {s^cs} 63a+.64a 1723.53 Mark: and see if it's preferable and if it is then we'll get more . {s^cs^2} 64b 1725.47 Mark: comfort . {s}

PROVIDE(justification) 1714.09 John: because there's no point in doing that if it's not going to be any better . {s} 61b+

Why was David’s proposal on microphones rejected?


Automa(c Argumenta(ve Analysis

•  Based on the GETARUNS system1. •  Clauses in Turns are labelled with Primi(ve Discourse Rela(ons: –  statement, narra,on, adverse, result, cause, mo,va,on, explana,on, ques,on, hypothesis, elabora,on, permission, incep,on, circumstance, obliga,on, evalua,on, agreement, contrast, evidence, hypoth, seCng, prohibi,on.

•  And then Turns are labelled with Argumenta(ve labels: –  ACCEPT, REJECT/DISAGREE, PROPOSE/SUGGEST, EXPLAIN/JUSTIFY, REQUEST EXPLANATION/JUSTIFICATION.

1 Delmonte R., Bistrot A., Pallo:a V.,Deep Linguis(c Processing with GETARUNS for spoken dialogue Understanding. Proceedings LREC 2010 (P31 Dialogue Corpora). © 2011 interAnaly(cs 10

Evalua(on

Correct Incorrect Total Found Precision

Accept 662 16 678 98%

Reject 64 18 82 78%

Propose 321 74 395 81%

Request 180 1 181 99%

Explain 580 312 892 65%

Total 1826 421 2247 81.26%

Precision: 81.26% Recall: 97.53%

ICSI corpus of mee(ngs (Janin et al., 2003)

Delmonte R., Bistrot A., Pallo:a V.,Deep Linguis(c Processing with GETARUNS for spoken dialogue Understanding. Proceedings LREC 2010 (P31 Dialogue Corpora).


EXPERIMENTS WITH CALL CENTER DATA


Ra(onale: implement the four objec(ves

1.  Iden(fy Customer Oriented Behaviors, 2.  Iden(fy Root Cause of Problems 3.  Iden(fy customers who need par(cular

a:en(on 4.  Learn best prac(ces in dealing with

customers


The Data

•  Corpus of 213 manually transcribed conversa(ons of a help desk call center in the banking domain.

•  Average of 66 turns per conversa(on. •  Average of 1.6 calls per agent. •  Collected for a study aimed at iden(fying customer oriented behaviors that could favor sa(sfactory interac(on with customers (Rafaeli et al. 2007).


Iden(fy Customer Oriented Behaviors

•  Based on the work of Rafaeli et al. 2006. •  Customer Oriented Behaviors – an(cipa(ng customers requests 22,45% – educa(ng the customer 16,91% – offering emo(onal support 21,57% – offering explana(ons / jus(fica(ons 28,57% – personaliza(on of informa(on 10,50%


Significant correla(on with argumenta(ve labels


Iden(fy Root Cause of Problems •  Coopera(veness score –  a measure obtained by averaging the score obtained by mapping argumenta(ve labels of each turn in the conversa(on into a [-‐5 +5] scale.

•  Sen(ment Analysis module.

Argumenta=ve Categories Coopera=veness

Accept explana(on 5

Suggest 4

Propose 3

Provide opinion 2

Provide explana(on/jus(fica(on 1

Request explana(on/jus(fica(on 0

Ques(on -‐1

Raise issue -‐2

Provide nega(ve opinion -‐3

Disagree -‐4

Reject explana(on or jus(fica(on -‐5


Top 20 Controversial Topics with average coopera(veness scores and sen(ment


Coopera(veness of speakers on top discussed topics


Iden(fy problema(c customers


Select a specific customer


Visualize a selected call


CONCLUSIONS


Conclusions •  New Genera(on Call Center Analy(cs requires Interac(on Mining –  Call Center Qualita(ve metrics and KPIs can be only implemented with a full understanding of the customer interac(on dynamics

•  Argumenta(on is pervasive in conversa(ons. –  In order to recognize argumenta(ve acts, advanced Natural Language Understanding is necessary.

•  Future work: –  Scalability: need to process millions of call per day! – Mul(-‐language: call centers all over the world.


The Team


23

Dr. Lammert Vrieling (1968) - Chief Executive Officer‣ 15 years in both profit and not-for-profit organizations as consultant, trainer/coach and as

executive. ‣ Experience in the steel and aluminium industry, multimedia publishing and newspaper,

financial services and in the not-for-profit sector.

Prof. Dr. Rodolfo Delmonte (1946) - Chief Science Officer‣ Since 1993 head of Computational Linguistics Laboratory of the University of

Venice, Italy ‣ 30+ years experience in computational linguistics‣ From 1986 to 1992 he worked with the Department of Engineering of the

University of Parma. From 1978 to 1986 worked with the Department of Dr. Vincenzo Pallotta (1966) - Chief Technology Officer‣ 30 years in ICT‣ 10 years in R&D‣ Human-Language Technology, Digital Libraries, Artificial Intelligence,

Ubiquitous Computing, Human-Computer Interaction, Usability Engineering, Information Retrieval, Web Search Engines, Semantic Web, Computational Logics, Training and Education, e-learning.

David E. Walker (1964) - Chief Operating Officer‣ 25+ years in IT as Software engineer, developer, project manager, and architect. ‣ Senior Software Solutions Architect with extensive experience in designing, developing and

delivering enterprise solutions for payment processing, human resource, healthcare, marketing, manufacturing and scientific research environments.

www.interanaly=cs.ch …find us at

Interaction Mining: the new frontier of Call Center Analytics

Technology

Transcript of Interaction Mining: the new frontier of Call Center Analytics