1

Emotion Classification Using Massive Examples Extracted from the Web

Ryoko Tokuhisa, Kentaro Inui, Yuji Matsumoto

Toyota Central R&D Labs/Nara Institute of Science and Technology

Coling 2008

2

Emotion Classification Given an input sentence, classify the

sentence into 10 emotion classes Need to construct an emotion-provoking

corpus (EP corpus) Example:

I was disappointed because the shop was closed and I’d traveled a long way to get there

Emotion: disappointment Event: the shop was closed and I’d traveled a

long way to get there

3

Building an EP corpus (1/8) Ten emotions: happiness,

pleasantness, relief, fear, sadness, disappointment, unpleasantness, loneliness, anxiety, anger

Built a hand-crafted lexicon of emotion words from the Japanese Evaluation Expression Dictionary

349 emotion words

4

Building an EP corpus (2/8)

5

Building an EP corpus (3/8)

A subordinate clause was extracted as an emotion-provoking event if (a) it was subordinated to a matrix

clause headed by an emotion word (b) the relation between the

subordinate the matrix clauses is marked by eight connectives (Japanese)

6

Building an EP corpus (4/8)

“It suddenly started raining” provokes disappointment

7

Building an EP corpus (5/8) Applying the lexical patterns to the

Japanese Web corpus, which contains 500 million sentences, 1.3 million events were collected

8

Building an EP corpus (6/8)

9

Building an EP corpus (7/8) An annotator evaluated 2000 randoml

y chosen events Correct: correct example Context-dep: context-dependent Error: error example

10

Building an EP corpus (8/8)

11

Sentiment Polarity Classification (1/5)

Finding neutral events is difficult Collected 1000 sentences

randomly from the web and investigate their polarity

12

Sentiment Polarity Classification (2/5) Two-step approach

First classify a given input into three sentiment polarity classes, either positive, negative or neutral

Then classify only those judged positive or negative into the 10 fine-grained emotion classes

13

Sentiment Polarity Classification (3/5) The sentiment classification model is

trained with SVMs Test sentence is neutral if the output

of the classification model is near the decision boundary

Features are 1-gram, 2-gram and 3-gram extracted from word-polarity lattice

14

Sentiment Polarity Classification (4/5)

Features: child, positive, child-of, positive-of, child-of-education, …

15

Sentiment Polarity Classification (5/5) Polarity value of each word is

defined in a sentiment dictionary, which includes 1880 positive words and 2490 negative words One annotator identified positive and

negative words from the 50 thousand most frequent words samples from the Web

16

Emotion Classification

Use k-nearest-neighbor approach (kNN)

17

Experiments: Sentiment polarity classification (1/2)

TestSet1: 6 subject speakers produce 31 positive utterances, 34 negative utterances, and 25 neutral utterances

TestSet2: Used the 1140 samples that were labeled Correct before 491 positive samples, 649 negative

samples Add 501 neutral samples from the Web

18

Experiments: Sentiment polarity classification (2/2)

19

Experiments: Emotion classification (1/3) TestSet1 (2p, best): Two annotators annotated

each positive or negative sentence in TestSet1 with exactly one of the 10 emotion classes. A model’s answer is correct if it was identical with either of the two labels

TestSet1 (1p, acceptable): One annotator is asked to annotate each positive or negative sentence in TestSet1 with all the emotions involved in it. A model’s answer was considered correct if it was identical with one of the labeled classes

TestSet2: Use the results from the evaluation of EP corpus’ quality

20

Experiments: Emotion classification (2/3) Baseline

Kozareva’s PMI unsupervised model Use Google search engine to obtain an e

motion and a word’s co-occurrence frequency

k-NN 1-NN, 3-NN, 10-NN

21

Experiments: Emotion classification (3/3)