Sentence Classifier Sentence Classifier for Helpdesk Emailsfor Helpdesk Emails

Anthony

6 June 2006

Supervisors: Dr. Yuval Marom Dr. David Albrecht

Outline of Topics

IntroductionDomainApproachFeature SelectionContext-based Sentence ClassificationExperiments and ResultsQuestion & Answer

Introduction

Increased usage of emails: communication, organizing workflow, managing tasksA need to process these emails to better organize themExisting tasks: Email Summarization Email Classification Spam filtering

Use of words-only

Sentence Types

Can they be useful for existing tasks?Depend on the domainExamples Invitation Instruction Suggestion Complaint

Domain

Domain: Email helpdeskSpecifically, email responses (more

structured) Availability of data Analyze the efficiency of email

responses

Sample Email<response-ack>This is with reference to your email

regarding `Service Pack 1’</response-ack>

<instruction>Download and install the latest service packs from the link provided </instruction>

<specification>Compaq insight Manager 7<specification>

<url> http://www.hp.com/ </url>

<request>Please email us in case of any queries</request>

Thesis Aim

Develop a sentence classifierFocus: Investigate several feature selection

methods Investigate the use of context in

sentence classification

Motivation

Applications Identify informative sentences to

summarize biography Classify sentences in online product

reviews Use important sentences to help

classify documents Use sentences in the email to

determine sender’s intention

Approach

Determine the sentence typesCreate training setClassification Methods: Naïve Bayes Decision Trees SVM

Class Proportion

Class %

Statement 28.5

Thanking 15.3

Request 9.8

Salutation 8.7

Instruction 8.5

Instruction-item 6.3

URL 5.4

Class %

Response-ack

4.2

Suggestion 3.7

Specification 2.8

Signature 2.2

Apology 1.5

Questions 1.5

Others 1.5

Feature Set

Sentences need to be transformed into an appropriate representation for most classification algorithmsCommon features: Bag-of-Words (“version”, “latest”,

“software”, “install”) Bigram, Trigram

(“thank you”, “we are sorry”)

Feature Selection Purpose

High feature space: (tens of) thousands of featuresA need to reduce the feature space for Computational efficiency Remove redundant features (possibly) Improve classification

accuracy

Feature Selection Methods

Feature selection methods: Stop-words removal (“of”, “the”, “a”) Lemmatization Sentence Frequency Information Gain Chi Square

Context-based Sentence Classification

Classify a sequence of sentences extracted from an emailContext of a sentence refers to its surrounding sentences

...Set serial speed at least 38.4K.Issue AT^H carriage return.Begin your ASCII file upload.…

Context-based Sentence Classification

Assume given the class of previous sentenceFind the upper bound of improvement

...

<instruction>Set the speed at least 38.4</instruction>

<instruction>Issue AT^H carriage return</instruction>

Begin your ASCII file upload

…

Evaluation Metrics (1)Evaluation for each categoryEvaluation for average of all categoriesCommon metrics Precision Recall F1-measure

Evaluation Metrics (2)

P

PA

A

PAPA

Precision =

Recall =

F1-measure =recallprecision

recallprecision

2

Experiment and Results

Classifier

F1-measure

Without Feature

Selection

With Feature

Selection

Naïve Bayes 0.666 0.814

Decision Trees

0.829 0.844

SVM 0.883 0.888

Effect of Feature Selection

Chi-Square Effect on Different Classifiers

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Number of features

F-m

easu

re NB

DT

SVM

Class-by-Class Analysis

0.00.1

0.20.3

0.40.50.6

0.70.8

0.91.0

Sentence Types

F-m

easu

re

NB

DT

SVM

Effect of Context

ClassifierF1-measure

Without Context With Context

Naïve Bayes 0.814 0.844

Decision Trees

0.844 0.846

SVM 0.888 0.864

Analysis on Context

Classifier Corrections

Misclassifications

Difference

Naïve Bayes 45 11 34

Decision Trees

8 6 2

SVM 25 11 14

...<instruction>Set the speed at least 38.4</instruction><instruction>Issue AT^H carriage return.</instruction><instruction>Begin your ASCII file upload.</instruction>…

Conclusion

Feature selection methods have positive effectSVM > Decision Trees > Naïve BayesContext shows minor improvement Need more data

Future Work

Parse Trees Consider the structure of the

sentence

Viterbi Algorithm Find the best sequence of classes to

map to the sequence of sentences

Forward-backward Algorithm Include next sentences as the context

to predict current sentence

Question and Answer