PACLIC 28

!185

Automatically Building a Corpus for Sentiment Analysis on IndonesianTweets

Alfan Farizki Wicaksono, Clara Vania, Bayu Distiawan T., Mirna AdrianiInformation Retrieval Lab.

Faculty of Computer Science, University of IndonesiaDepok, Republic of Indonesia

{alfan, c.vania, b.distiawan, mirna}@cs.ui.ac.id

Abstract

The popularity of the user generated content,such as Twitter, has made it a rich sourcefor the sentiment analysis and opinion min-ing tasks. This paper presents our study inautomatically building a training corpus forthe sentiment analysis on Indonesian tweets.We start with a set of seed sentiment corpusand subsequently expand them using a classi-fier model whose parameters are estimated us-ing the Expectation and Maximization (EM)framework. We apply our automatically builtcorpus to perform two tasks, namely opin-ion tweet extraction and tweet polarity clas-sification using various machine learning ap-proaches. Experiment result shows that a clas-sifier model trained on our data, which isautomatically constructed using our proposedmethod, outperforms the baseline system interms of opinion tweet extraction and tweetpolarity classification.

1 Introduction

There are millions of textual messages or posts gen-erated by internet users everyday on various usergenerated content platfroms, such as microblogs(e.g. Twitter1), review websites, and internet fo-rums. They post about their stories, experiences,current events that are happening, as well as opin-ions about products. As a result, the user generatedcontent has become a rich source for mining usefulinformation about various topics.

Twitter, one the popular microblogging platforms,is currently getting a lot of attention from internet

1http://twitter.com

users because it allows users to easily and instantlypost their thoughts of various topics. Twitter cur-rently has over 200 million active users and produce400 million posts each day 2. The posts, known astweets, often contain useful knowledge so that manyresearchers focus on Twitter for conducting NLP-related research. McMinn et al. (2014) harnessedmillions of tweets to develop an application for de-tecting, tracking, and visualizing events in real-time.Previously, Sakaki et al. (2013) also used twitteras a sensor for earthquake reporting system. Theyclaimed that the system can detect an earthquakewith high probability merely by monitoring tweetsand the notification can be delivered faster thanJapan Meteorology Agency announcements. More-over, Tumasjan et al. (2010) demostrated that Twit-ter can also be used as a resource for political fore-casting.

Due to the nature of Twitter, tweets usually ex-press peoples personal thoughts or feelings. There-fore, tweets serve as good resources for sentimentanalysis and opinion mining tasks. Many compa-nies can benefit from tweets to know how many pos-itive responses and/or negative responses towardstheir products as well as the reasons why consumerslike/dislike their products. They can also leveragetweets to gain a lot of insight about their competi-tors. Consumers can also use information fromtweets regarding the quality of a certain product.They commonly learn from peoples past experienceswho have already used the product before they de-cide to purchase it. To realize the aforementioned

2https://blog.twitter.com/2013/celebrating-twitter7

Copyright 2014 by Alfan Farizki Wicaksono, Clara Vania, Bayu Distiawan T., and Mirna Adriani28th Pacific Asia Conference on Language, Information and Computation pages 185–194

PACLIC 28

!186

ideas, many researchers have put a lot of effort totackle one of the important tasks on Twitter senti-ment analysis, that is, tweet polarity classification(Nakov et al., 2013; Hu et al., 2013; Kouloumpis etal., 2011; Agarwal et al., 2011; Pak and Paroubek,2010). They proposed various approaches to deter-mine whether a given tweet expresses positive ornegative sentiment.

In this paper, we address the problem of sentimentanalysis on Indonesian tweets. Indonesian languageitself currently has more than 240 millions of speak-ers spread in mostly areas of south-east asia. Inaddition, Semiocast, a company who provides dataintelligence and research on social media, has re-vealed that Indonesia ranked 5th in terms of Twit-ter accounts in July 2012 and users from Jakartacity (i.e. capital city of Indonesia) were the mostactive compared to the users from other big cities,such as Tokyo, London, and New York 3. Therefore,there is absolutely a great need for natural languageprocessing research on Indonesian tweets, especiallysentiment analysis, since there would be a lot of in-formation which is worth obtaining for many pur-poses. Unfortunately, Indonesian language is cat-egorized as an under-resourced language because itstill suffers from a lack of basic resources (especiallylabeled dataset) needed for a various language tech-nologies.

There are two tasks addressed in this paper,namely opinion tweet extraction and tweet polarityclassification. The former task is aimed at selectingall tweets comprising users’ opinion towards some-thing and the latter task is to determine the polaritytype of an opinionated tweet (i.e., positive or neg-ative tweet). To tackle the aforementioned tasks,we employ machine learning approach using train-ing data and word features. However, a problemthen appears when we do not have annotated datato train our models. Asking people to manually an-notate thousands, even millions of tweets with highquality is not our choice since it is very expensiveand time-consuming due to the massive scale andrapid growth of Twitter.

To overcome the aforementioned problem, wepropose a method that can automatically develop

3http://semiocast.com/en/publications/2012_07_30_Twitter_reaches_half_a_billion_accounts_140m_in_the_US

training data from a pool of millions of tweets. First,we automatically construct a small set of labeledseed corpus (i.e. small collection of positive andnegative tweets) that will be used for expanding thetraining data in the next step. Next, we expand thetraining data using the previously constructed seedcorpus. To do that, we use the rationale that senti-ment can be propagated from the labeled seed tweetsto the other unlabeled tweets when they share similarword features, which means that the sentiment typeof an unlabeled tweet can be revealed based on itscloseness to the labeled tweets. Based on that idea,we employ a classifier model whose parameters areestimated using labeled and unlabeled tweets via Ex-pectation and Maximization (EM) framework. Inthis method, we incorporate two types of dataset: thefirst dataset is a small set of labeled seed tweets andthe second dataset is a huge set of unlabeled tweetsthat serve as a source for expanding the training data.Intuitively, this method allows us to propagate senti-ment from labeled tweets to unlabeled tweets. Later,we show that the training data automatically con-structed by our method can be used by the classifiersto effectively tackle the problem of opinion tweet ex-traction and tweet polarity classification.

In summary, the main contributions of this paperis two-folds: first, we present a method to automati-cally construct training instances for sentiment anal-ysis on Indonesian tweets. Second, we show somesignificant works for sentiment analysis on Indone-sian tweets which were rarely addressed before.

2 Related Works

There have been extensive works on opinion min-ing and sentiment analysis as described in (Pang andLee, 2008). They presented various approaches andgeneral challenges to develop applications that canretrieve opinion-oriented information. Moreover,Liu (2007) clearly mentions the definition of opin-ionated sentence as well as describes two sub-tasksrequired to perform sentence-level sentiment analy-sis, namely, subjectivity classification and sentence-level sentiment classification. However, previousresearchers primarily focused on performing senti-ment analysis on review data. The trends has shiftedrecently when social networking platform, such asFacebook and Twitter, has been growing rapidly. As

PACLIC 28

!187

a result, many researchers has now started to per-form sentiment analysis on microblogging platform,such as twitter (Hu et al., 2013; Nakov et al., 2013;Kouloumpis et al., 2011; Pak and Paroubek, 2010).In our work, we perform two-level sentiment analy-sis, similar to that described in (Liu, 2007). In addi-tion, we also perform sentiment analysis on tweets(i.e. Indonesian tweets), instead of general sen-tences.

Current sentiment analysis research mostly re-lies on manually annotated training data (Nakovet al., 2013; Agarwal et al., 2011; Jiang et al.,2011; Bermingham and Smeaton, 2010). However,employing humans for manually annotating thou-sands, even millions of tweets is absolutely labor-intensive, time-consuming, and very expensive dueto the massive scale and rapid growth of Twitter.This becomes a significant obstacle for researcherswho want to perform sentiment analysis on tweetsposted in under-resourced language, such as Indone-sian tweets. Limited works have been done previ-ously on automatically collecting training data (Pakand Paroubek, 2010; Bifet and Frank, 2010; Davi-dov et al., 2010). Some researchers harnessed happyemoticons and sad emoticons to automatically col-lect training data (Pak and Paroubek, 2010; Bifetand Frank, 2010). They assumed that tweets con-taining happy emoticons (e.g. ”:)”, ”:-)”) have posi-tive sentiment, and tweets containing sad emoticons(e.g. ”:(”, ”:-(”) have negative sentiment. Unfortu-nately, their method clearly cannot get the coverageto reach sentiment-bearing tweets as many as pos-sible since not all sentiment-bearing tweets containemoticons.

Limited attempts have been made to perform sen-timent analysis on Indonesian tweets. Calvin andSetiawan (2014) performed tweet polarity classi-fication limited to the tweets talking about tele-phone provider companies in Indonesia. Their clas-sification method relies on a small set of domain-dependent opinionated words. Before that, Aliandu(2014) conducted research on classifying an Indone-sian tweet into three classes: positive, negative, andneutral. Aliandu (2014) used the method proposedby Pak and Paroubek (2010) to collect training data,that is, emoticons for collecting sentiment-bearingtweets. Even though those researchers performedsimilar works to us, we have two different points.

First, we use different techniques to automaticallycollect training data. Second, we perform two-levelsentiment analysis, namely, opinion tweet extractionand tweet polarity classification. Moreover, in theexperiment section, we show that our method to col-lect training data is better than the one proposed byPak and Paroubek (2010). Our method also pro-duces much larger data since we do not rely on sheeremoticon-containing tweets to collect training data.

3 Automatically Building Training Data

3.1 Data CollectionOur corpus consists of 5.3 million tweets whichwere collected using Twitter Streaming API be-tween May 16th, 2013 and June 26th, 2013. Aswe wanted to build Indonesian sentiment corpus, weused tweet’s geo-location to filter tweets posted inthe area of Indonesia. We applied language filteringbecause based on our observation, Indonesian Twit-ter users also like to use English or local languagein their tweets. We then divided our corpus into fourdisjoint datasets. Table 1 shows the overall statisticsof our Twitter corpus.

Dataset Label #Tweets

DATASET1 Unlabeled 4,291,063DATASET2 Unlabeled 1,000,000DATASET3 Neutral 12,614DATASET4 Pos, Neg, Neutral 637

Total 5,304,314

Table 1: The statistics of our Tweet collection

To collect DATASET3 (i.e. neutral or non-opinion tweets), we used the same approach as in(Pak and Paroubek, 2010). First, we selected somepopular Indonesian news portal accounts from theoverall corpus and then labeled them as objective.Here, we assume that tweets from news portal ac-counts are neutral as it usually comes from headlinenews. This method was actually proposed by (Pakand Paroubek, 2010). But, we also did some empir-ical observation and acknowledged that this methodperforms quite well to collect neutral tweets.

The remaining corpus which is not pub-lished by news portal accounts is then used tobuild seed corpus (DATASET2), development cor-pus (DATASET1), and gold-standard testing data

PACLIC 28

!188

(DATASET4). In this study, DATASET2 is used toconstruct labeled seed corpus. The seed corpus itselfcontains initial data that is believed to have opinionas well as sentiment. On the other side, develop-ment corpus DATASET1 contains unlabeled tweetsused to expand our seed corpus. Our testing data(DATASET4) consists of 637 tweets which weretagged manually by the human annotators. Thesetweets were collected using some topic words whichhave tendency to be discussed by a lot of people.Two annotators were asked to independently classifyeach tweet into three classes: positive, negative, andneutral. The agreement of the annotators reachedthe level of Kappa value 0.95, which is consideredas a satisfactory agreement. The label of each tweetin DATASET4 is the label agreed by the two anno-tators. But, when they did not agree, we asked thethird annotators to decide the label. It is also worthto note that our testing data comes from various do-mains, such as telephone operator, public transporta-tion, famous people, technology, and films. Table 2and 3 shows the details of DATASET4.

Sentiment Type #Tweets

Positive 202Negative 132Neutral 303Total 637

Table 2: The statistics of DATASET4

Domain #Tweets

Telephone operators 94Public transportations 53

Government companies 11Figures/People 61Technologies 12

Sports and Athletes 41Actress 29Films 67

Food and Restaurants 34News 214Others 21Total 637

Table 3: The domains in DATASET4

We also show some examples of Tweets found in

DATASET4 as follows:

• ” Telkomsel memang wokeeehhh (free internet):)” (Telkomsel is nice (free internet) :))

• ”Kecewa sama trans Jakarta. Manajemen blmbagus. Masa hrs nunggu lbh dr 30 menit utknaek busway.” (really dissapointed in trans-jakarta. The management is not good. Wewaited for more than 30 minutes to get the buson)

• ”man of steel keren bangeeeettttt :D” (Man ofsteel is really cool :D)

• ”RT @detikcom: Lalin Macet, Pohon Tumbangdi Perempatan Cilandak-TB Simatupang” (RT@detikcom: Traffic jam, a tree tumbled downin the Cilandak-TB Simatupang intersection)

3.2 Building Seed Training InstancesAs we explained before, our seed corpus containsinitial data used for expanding the training corpus.We propose two automatic techniques to constuctthe seed corpus from DATASET2:

3.2.1 Opinion Lexicon based TechniqueIn the first technique, we use Indonesian opinion

lexicon (Vania et al., 2014) to construct our seed cor-pus. A tweet will be classified as positive if it con-tains more positive words then negative words andvice versa. If a tweet contains word with a particu-lar sentiment but the word is preceded by a negation,the polarity of the tweet will be shifted to its oppositesentiment. Moreover, we did not consider the tweetsthat do not contain any words from the opinion lex-icon. In total, we have collected 135,490 positiveseed tweets and 99,979 negative seed tweets.

3.2.2 Clustering based TechniqueThe second technique was implemented by us-

ing clustering (Li and Liu, 2012). This techniquehas several advantages, such as we do not need toprovide any resources, such as lexicon or dictio-nary for a particular language. Each tweet fromDATASET2 will be put into three clusters, namelypositive tweets, negative tweets, or neutral tweets.We use all terms and POS tags from the tweet andeach term is weighted using the TF-IDF as a fea-tures. Using this approach, 194 tweets were grouped

PACLIC 28

!189

into negative tweets, 325 tweets were grouped intopositive tweets, and the rest was left out.

3.3 Adding New Training Instances

After we automatically construct labeled seed cor-pus from DATASET2, we are now ready to ob-tain more training instances. We use DATASET1,which is much bigger than DATASET2, as a sourcefor expanding training data. The idea is that senti-ment scores of all unlabeled tweets in DATASET2can be revealed using propagation from labeled seedcorpus. To realize that idea, we employ a clas-sifier model whose parameters are estimated us-ing labeled and unlabeled tweets via Expectationand Maximization (EM) framework. The well-known research done by (Nigam et al., 2000) haveshown that Expectation and Maximization frame-work works well for expanding training data totackle the document-level text classification prob-lem. In our work, we also show that this frameworkworks quite well for tweets.

EM algorithm is an iterative algorithm for find-ing maximum likelihood estimates or maximum aposteori estimates for models when the data is in-complete (Dempster et al., 1977). Here, our data isincomplete since the sentiment scores of unlabeledtweets are unknown. To reveal the sentiment scoresof unlabeled tweets using EM algorithm, we per-form several iterations. First, we train the classi-fier with just the labeled seed corpus. Second, weuse the trained classifier to assign probabilistically-weighted labels or sentiment scores (i.e. the proba-bility of being a positive and negative tweet) to eachunlabeled tweets. Third, we trained once again themodel using all tweets (i.e. both the originally andnewly labeled tweets). These last two steps are iter-ated until the parameters of the model do not change.At each iteration, the sentiment scores of each unla-beled tweets are improved as the likelihood of theparameters is guaranteed to improve until there is nomore change (Dempster et al., 1977). In addition,only tweets whose sentiment scores surpass a cer-tain threshold will be considered as our new traininginstances.

Formally, we have a set of tweets T divided intotwo disjoint partitions: a set of labeled seed tweets T

l

and a set of unlabeled tweets Tu

, such that T = Tl

[Tu

. In this case, Tl

represents seed tweets which are

selected from DATASET2 and automatically labeledusing the method described in the previous sectionand T

u

represents a set of all tweets in DATASET1.Each tweet t

i

2 T , that has length |ti

|, is defined asan ordered list of words (w1, w2, ..., w|V |) and eachword w

k

is an element of the vocabulary set V =

{w1, w2, ..., w|V |}.For the classifier in the iteration, we employ

Naive Bayes classifier model. In our case, given atweet t

i

and two class label Cj

, where j 2 S andS = {pos, neg}, the probability that each of thetwo component classes generated the tweet is deter-mined using the following equation:

P (C

j

|ti

) =

P (C

j

)

Q|ti|k=1 P (w

k

|Cj

)

Pj2S P (C

j

)

Q|ti|k=1 P (w

k

|Cj

)

(1)

The above equation holds since we assume thatthe probability of a word occuring within a tweetis independent of its position. Here, the collectionof models parameters, denoted as ✓, is the collec-tion of word probabilities P (w

k

|Cj

) and the classprior probabilities P (C

j

). Given a set of tweet data,T = {t1, t2, ..., t|T |}, the Naive Bayes uses the max-imum a posteori (MAP) estimation to determine thepoint estimate of ✓, denoted by b

✓. This can be doneby finding ✓ that maximize P (✓|T ) / P (T |✓)P (✓).This yields the following estimation formulas foreach component of the parameter.

The word probabilities P (w

k

|Cj

) are estimatedusing the following formula:

P (w

k

|Cj

) =

1+P|T |

i=1 N(wk,ti).P (Cj |ti)|V |+

P|V |n=1

P|T |i=1 N(wn,ti).P (Cj |ti)

(2)

where N(w

k

, t

i

) is the number of occurences ofword w

k

in tweet ti

. Similarly, the class prior prob-abilities P (C

j

) are also estimated using the samefashion.

P (C

j

) =

1 +

P|T |i=1 P (C

j

|ti

)

|S|+ |T | (3)

In the above equation, P (C

j

|ti

), j 2 {pos, neg},are sentiment scores associated with each tweet t

i

2T , where

Pj

P (C

j

|ti

) = 1. For the labeled seedtweets, P (C

j

|ti

) are rigidly assigned since the labelis already known in advance:

PACLIC 28

!190

P (C

j

|ti

) =

⇢1 if t

i

belongs to class Cj

0 otherwise(4)

Meanwhile, for the set of unlabeled tweets T

u

,P (C

j

|ti

) are probabilistically assigned in each iter-ation, so that 0 P (C

j

|ti

) 1. Thus, the prob-ability of all the tweet data given the parameters,P (T |✓), is determined as follows:

P (T |✓) =Y

ti2T

X

j

P (t

i

|Cj

)P (C

j

) (5)

Finally, we can compute the log-likelihood of the pa-rameters, logL(✓|T ), using the following equation:

logL(✓|T ) ⇡ logP (T |✓)=

Pti2T log

Pj

P (t

i

|Cj

)P (C

j

)

(6)The last equation contains ”log of sums”, which

is difficult for maximization process. Nigam et al.(2000) shows that the lower bound of the last equa-tion can be found using Jensen’s inequality. As aresult, we can express the complete log-likelihoodof the parameters, logL

c

(✓|T ), as follows:

logL(✓|T )

� logL

c

(✓|T )

⇡P

ti2TP

j

P (C

j

|ti

) log(P (t

i

|Cj

)P (C

j

))

(7)The last equation is used in each iteration to

check whether or not the parameters have con-verged. When the EM iterative procedure ends dueto the convergence of the parameters, we then needto select several tweets from the set of unlabeledtweets T

u

, which are eligible for our new traininginstances. The criteria of selecting new training in-stances, denoted by T

n

, is as follows:

Tn

= {t 2 T

u

| |P (C

pos

|t)� P (C

neg

|t)| � ✏} (8)

where ✏ is an empirical value, 0 ✏ 1. Inour experiment, we set ✏ to 0.98 since we want toobtain very polarized tweets in terms of sentimentas our new training instances. In summary, the EMalgorithm for expanding training data is described asfollows:

• Input: A set of labeled seed tweets Tl

, and alarge set of unlabeled tweets T

u

• Train a Naive Bayes classifier using only the la-beled seed teets T

l

. The estimated parameters,b✓, are obtained using equation 2 and 3.

• Repeat until logLc

(✓|T ) does not change (i.e.the parameters do not change):

– [E-Step] Use the current classifier, b✓, to

probabilistically label all unlabeled tweetsin T

u

, i.e. we use equation 1 to obtainP (C

j

|ti

) for all ti

2 Tu

.– [M-Step] Re-estimate the parameters of

current classifier using all tweet data Tu

[Tl

(i.e. both the originally and newly la-beled tweets). Here, we once again usethe equation 2 and 3.

• Select the additional training instances, Tn

, us-ing the criteria mentioned in formula 8.

• Output: The expanded training data Tn

[ Tl

4 Experiments and Evaluations

4.1 Training Data Construction

After we applied our training data constructionmethod, we collected around 2.8 millions of opiniontweets when we used opinion lexicon based tech-nique to automatically construct labeled seed cor-pus. Meanwhile, when we used clustering basedtechnique to construct labeled seed corpus, wecollected around 2.4 millions of opinion tweets.We refer to the former yielded training datasetas LEX-DATA and the latter as CLS-DATA. Ta-ble 4 and 5 show the statistics of LEX-DATA andCLS-DATA, respectively.

Sentiment Type Pos Neg

#Seed Tweets 135,490 99,797#Added Tweets 1,180,506 1,419,438

Total 1,315,996 1,519,235

Table 4: The statistics of LEX-DATA

We also automatically collected training data us-ing the method proposed by Pak and Paroubek(2010). We used the well-known positive/negative

PACLIC 28

!191

Sentiment Type Pos Neg

#Seed Tweets 325 194#Added Tweets 1,332,741 1,160,387

Total 1,333,066 1,160,581

Table 5: The statistics of CLS-DATA

emoticons in Indonesian tweets, such as ”:)”, ”:-)”, ”:(”, ”:-(”, to capture the opinion tweets fromDATASET1 and DATASET2. We refer to this train-ing dataset as EMOTDATA, and we used it for com-parison to our proposed method. Table 6 shows thedetail of EMOTDATA.

Sentiment Type Pos Neg#Tweets 276,970 103,740

Table 6: The statistics of EMOTDATA

4.2 Evaluation Methodology

To evaluate our automatic corpus constructionmethod, we performed two tasks, namely opiniontweet extraction and tweet polarity classification,harnessing our constructed training data. In otherwords, we see whether or not a classifier modeltrained on our constructed training data is able topeform both the aforementioned tasks with high per-formance.

Task 1 - Opinion Tweet Extraction: Given a col-lection of tweets T, the task is to discover all opiniontweets in T. Liu (2011) defined an opinion as a posi-tive or negative view, attitude, emotion, or appraisalabout an entity or an aspect of the entity. Thus, weadapt the aforementioned definition for the opiniontweet.

Task 2 - Tweet Polarity Classification: The task isto determine whether each opinion tweet extractedfrom the first task is positive or negative.

To measure the performance of the classifier,we tested the classifier on our gold-standard set,i.e. DATASET4, which was manually annotatedby two people. In addition, we also compared ourmethod against the method proposed by Pak andParoubek (2010). For the classifier, we employ two

well-known classifier algorithms, namely the NaiveBayes classifier and the Maximum Entropy model(Berger et al., 1996). We use the unigrams as ourfeatures, i.e. the presence of a word and its fre-quency in a tweet, since unigrams provide a goodcoverage of the data and most likely do not suf-fer from the sparsity problem. Morever, Pang et al.(2002) previously had shown that unigrams serve asgood features for sentiment analysis.

Before we train our classifier models, we applydata preprocessing process to all datasets. This isdone because tweets usually contain many informalforms of text that can be difficult to be recognizedby our classifiers. We use the following data prepro-cessing steps to our training data:

• Filtering: we remove URL links, Twitter useraccounts (started with ’@’), retweet (RT) infor-mation, and punctuation marks. All tweets arenormalized to lower case and repeated charac-ters are replaced by a single character.

• Tokenization: we split each tweet based onwhitespaces.

• Normalization: we replace each abbreviationfound in each tweet with its actual meaning.

• Handling negation: each negation term is at-tached to a word that follows it.

4.3 Evaluations on Opinion Tweet Extraction

As we mentioned previously, we see the problemof opinion tweet extraction as a binary classifica-tion problem. Thus, we assume that a tweet canbe classified into two categories: an opinion tweetand non-opinion tweet. For the testing data, we useDATASET4 that consists of 303 neutral/non-opiniontweets and 334 opinion tweets (i.e. the combina-tion of positive and negative tweets). For the train-ing data, we only have 12,614 non-opinion tweetsfrom DATASET3. But, we have a larger set of opin-ion tweets either from LEX-DATA, CLS-DATA, orEMOTDATA depending on the method we apply.To cope with this problem, we randomly selected12,614 opinion tweets either from LEX-DATA,CLS-DATA, or EMOTDATA so that the training datais balanced. Moreover, we use the precision, recall,and F1-score as our evaluation metrics.

PACLIC 28

!192

First, we measured the performance of the classi-fiers trained on the data constructed by the methodproposed by Pak and Paroubek (2010). We re-fer to this method as BASELINE. Furthermore,the non-opinion training data consists of all tweetsin DATASET3 and the opinion training data con-sists of 12,614 tweets randomly selected fromEMOTDATA. Second, we evaluated the classifierstrained on the data constructed using our proposedmethod. In this case, we run experiment usingthe two different seed corpus construction tech-niques. We refer to the method that use clusteringbased technique (for constructing seed corpus) asCLS-METHOD and the method that use opinion lexi-con as LEX-METHOD. The opinion training data wasconstructed in the same manner as before. This time,we used LEX-DATA and CLS-DATA to randomlyselect 12,614 opinion tweets for LEX-METHOD andCLS-METHOD, respectively.

Model Prec(%) Rec(%) F1(%)BASELINE

Naive Bayes 75.47 58.98 66.21Maxent 78.36 74.85 76.56

LEX-METHODNaive Bayes 76.24 64.37 69.80

Maxent 81.90 79.94 80.91CLS-METHOD

Naive Bayes 73.11 46.40 56.77Maxent 80.00 63.47 70.78

Table 7: The evaluation results for opinion Tweet extrac-tion task

Table 7 shows the results of the experiment. Wecan see that the classifiers trained on EMOTDATA,which was constructed using BASELINE, actuallyperform quite well. Maximum Entropy modelachived 76,56% in terms of F1-score, which is farfrom the performance score resulting from NaiveBayes model. It is worth to note that the clas-sifiers trained on LEX-DATA outperform thosetrained on EMOTDATA by over 3% and 4% forNaive Bayes and Maximum Entropy model, respec-tively, which means that LEX-METHOD is betterthan BASELINE. But, the situation is different forCLS-METHOD. This is actually no surprise sinceLEX-METHOD uses a good prior knowledge ob-tained from opinion lexicon. This might also suggest

that the seed corpus construction is an important as-pect in our method.

4.4 Evaluations on Tweet PolarityClassification

After we extract the opinion tweets, we then clas-sify the sentiment type of the opinion tweets intotwo classes: positive and negative. In the first sce-nario, we evaluated the classifiers trained on bothpositive and negative tweets from EMOTDATA sincewe aimed at comparing BASELINE against our pro-posed method. In the second scenario, we thenmeasured the performance of the classifiers whenthey were trained on the data constructed by ourmethod (i.e. LEX-METHOD and CLS-METHOD).For the testing data, both scenarios use DATASET4that consists of 202 positive tweets and 132 negativetweets. We left the neutral/non-opinion tweets. Forthe training data, the first scenario uses all tweets inEMOTDATA as the training data. But, we cannot di-rectly use all tweets in LEX-DATA or CLS-DATAfor the second scenario since the size of LEX-DATAand CLS-DATA, respectively, is much bigger thanEMOTDATA. As a result, due to fairness, we ran-domly selected 276,970 positive tweets and 103,740negative tweets from LEX-DATA and CLS-DATA,respectively, and subsequently use them for the sec-ond scenario. Moreover, we use a classification ac-curacy as our metric in this experiment.

Model Accuracy(%)BASELINE

Naive Bayes 74.85Maxent 73.35

LEX-METHODNaive Bayes 81.13

Maxent 86.82CLS-METHOD

Naive Bayes 42.81Maxent 45.80

Table 8: The evaluation results for Tweet polarity classi-fication task

Table 8 shows the results. We can see thatthe classifiers trained on LEX-DATA significantlyoutperform those trained on EMOTDATA by over7% and 13% for Naive Bayes and Maximum En-tropy model, respectively. Just like the previ-

PACLIC 28

!193

Figure 1: The effect of training data size. Here, we usedthe training data constructed using LEX-METHOD

ous experiment, CLS-METHOD is no better thanLEX-METHOD and BASELINE. We also suggestthat Maximum Entropy model is a good model forour sentiment analysis task since the results showthat this model is mostly superior to Naive Bayesmodel.

We further investigated the effect of increasing thesize of training dataset on the accuracy of the clas-sifiers. In this case, we only examined LEX-DATAsince LEX-METHOD yielded the best result before.Figures 1 shows the results. Training data of size Nmeans that we use N/2 positive tweets and N/2 neg-ative tweets as the training instances. As we can see,learning from large training data plays an importantrole in tweet polarity classification task. But, wealso notice a strange case. When the size of trainingdata is increased at the last point, the performanceof Naive Bayes significantly drops. This should notbe the case for Naive Bayes. We admit that the qual-ity of our training data set is far away from perfectsince it is automatically constructed. As a result, ourtraining data set is still prone to noise disturbanceand we guess that this is why the performance ofNaive Bayes drops at the last point.

5 Conclusions and Future Works

We propose a method to automatically constructtraining instances for sentiment analysis and opinionmining on Indonesian tweets. First, we automati-cally build a set of labeled seed corpus using opinion

lexicon based technique and clustering based tech-nique. Second, we harness the labeled seed cor-pus to obtain more training instances from a hugeset of unlabeled tweets by employing a classifiermodel whose parameters are estimated using the EMframework. For the evaluation, we test our automat-ically built corpus on the opinion tweet extractionand tweet polarity classification tasks.

Our experiment shows that our proposed methodoutperforms the baseline system which merely usesemoticons as the features for automatically buildingthe sentiment corpus. When we tested on the opin-ion tweet extraction and tweet polarity classificationtasks, the classifier models trained on the trainingdata using our proposed method was able to extractopinionated tweets as well as classify tweets polar-ity with high performance. Moreover, we found thatthe seed corpus construction technique is an impor-tant aspect in our method since the evaluation showsthat prior knowledge from the opinion lexicon canhelp building better training instances than just us-ing clustering based technique.

In the future, this corpus can be used as one ofthe basic resources for sentiment analysis task, es-pecially for Indonesian language. For the sentimentanalysis task itself, it will be interesting to inves-tigate various features beside unigram that may beuseful in detecting sentiment on Indonesian Twittermessages.

ReferencesApoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow,

and Rebecca Passonneau. 2011. Sentiment analysisof twitter data. In Proceedings of the Workshop onLanguages in Social Media, LSM ’11, pages 30–38,Stroudsburg, PA, USA. Association for ComputationalLinguistics.

Paulina Aliandu. 2014. Sentiment analysis on indone-sian tweet. In The Proceedings of The 7th ICTS.

Adam L. Berger, Vincent J. Della Pietra, and StephenA. Della Pietra. 1996. A maximum entropy approachto natural language processing. Comput. Linguist.,22(1):39–71, March.

Adam Bermingham and Alan F. Smeaton. 2010. Clas-sifying sentiment in microblogs: Is brevity an advan-tage? In Proceedings of the 19th ACM InternationalConference on Information and Knowledge Manage-ment, CIKM ’10, pages 1833–1836, New York, NY,USA. ACM.

PACLIC 28

!194

Albert Bifet and Eibe Frank. 2010. Sentiment knowl-edge discovery in twitter streaming data. In Proceed-ings of the 13th International Conference on Discov-ery Science, DS’10, pages 1–15, Berlin, Heidelberg.Springer-Verlag.

Calvin and Johan Setiawan. 2014. Using text miningto analyze mobile phone provider service quality (casestudy: Social media twitter). International Journal ofMachine Learning and Computing, 4(1), February.

Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010.Enhanced sentiment learning using twitter hashtagsand smileys. In Proceedings of the 23rd InternationalConference on Computational Linguistics: Posters,COLING ’10, pages 241–249, Stroudsburg, PA, USA.Association for Computational Linguistics.

A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977.Maximum likelihood from incomplete data via the emalgorithm. JOURNAL OF THE ROYAL STATISTICALSOCIETY, SERIES B, 39(1):1–38.

Xia Hu, Lei Tang, Jiliang Tang, and Huan Liu. 2013. Ex-ploiting social relations for sentiment analysis in mi-croblogging. In Proceedings of the Sixth ACM Inter-national Conference on Web Search and Data Mining,WSDM ’13, pages 537–546, New York, NY, USA.ACM.

Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and TiejunZhao. 2011. Target-dependent twitter sentiment clas-sification. In Proceedings of the 49th Annual Meet-ing of the Association for Computational Linguistics:Human Language Technologies - Volume 1, HLT ’11,pages 151–160, Stroudsburg, PA, USA. Associationfor Computational Linguistics.

Efthymios Kouloumpis, Theresa Wilson, and JohannaMoore. 2011. Twitter sentiment analysis: The goodthe bad and the omg! In Lada A. Adamic, Ricardo A.Baeza-Yates, and Scott Counts, editors, ICWSM. TheAAAI Press.

Gang Li and Fei Liu. 2012. Application of a clusteringmethod on sentiment analysis. J. Inf. Sci., 38(2):127–139, April.

Bing Liu. 2007. Web Data Mining: Exploring Hyper-links, Contents, and Usage Data. Data-Centric Sys-tems and Applications. Springer.

Andrew J. McMinn, Daniel Tsvetkov, Tsvetan Yor-danov, Andrew Patterson, Rrobi Szk, Jesus A. Ro-driguez Perez, and Joemon M. Jose. 2014. An interac-tive interface for visualizing events on twitter. In Pro-ceedings of the 37th International ACM SIGIR Confer-ence on Research & Development in InformationRetrieval, SIGIR ’14, pages 1271–1272, New York,NY, USA. ACM.

Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva,Veselin Stoyanov, Alan Ritter, and Theresa Wilson.

2013. Semeval-2013 task 2: Sentiment analysis intwitter. In Second Joint Conference on Lexical andComputational Semantics (*SEM), Volume 2: Pro-ceedings of the Seventh International Workshop on Se-mantic Evaluation (SemEval 2013), pages 312–320,Atlanta, Georgia, USA, June. Association for Compu-tational Linguistics.

Kamal Nigam, Andrew Kachites McCallum, SebastianThrun, and Tom Mitchell. 2000. Text classifica-tion from labeled and unlabeled documents using em.Mach. Learn., 39(2-3):103–134, May.

Alexander Pak and Patrick Paroubek. 2010. Twit-ter as a corpus for sentiment analysis and opinionmining. In Proceedings of the Seventh conferenceon International Language Resources and Evaluation(LREC’10), Valletta, Malta, May. European LanguageResources Association (ELRA).

Bo Pang and Lillian Lee. 2008. Opinion mining andsentiment analysis. Found. Trends Inf. Retr., 2(1-2):1–135, January.

Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan.2002. Thumbs up?: Sentiment classification usingmachine learning techniques. In Proceedings of theACL-02 Conference on Empirical Methods in NaturalLanguage Processing - Volume 10, EMNLP ’02, pages79–86, Stroudsburg, PA, USA. Association for Com-putational Linguistics.

T. Sakaki, M. Okazaki, and Y. Matsuo. 2013. Tweetanalysis for real-time event detection and earthquakereporting system development. Knowledge and DataEngineering, IEEE Transactions on, 25(4):919–931,April.

A. Tumasjan, T.O. Sprenger, P.G. Sandner, and I.M.Welpe. 2010. Predicting elections with twitter: What140 characters reveal about political sentiment. InProceedings of the Fourth International AAAI Confer-ence on Weblogs and Social Media, pages 178–185.

Clara Vania, Mohammad Ibrahim, and Mirna Adriani.2014. Sentiment lexicon generation for an under-resourced language. International Journal of Compu-tational Linguistics and Applications (IJCLA) (To Ap-pear).