Text Mining Infrastructure in R

Presented ByAshraf Uddin

(http://ashrafsau.blogspot.in/)South Asian University, New Delhi, India.

29 January 2014

What is R?

A free software environment for statistical computing and graphics.

open source package based developed by Bell Labs

Many statistical functions are already built in

Contributed packages expand the functionality to cutting edge research

Implementation languages C, Fortran

What is R?

R is the result of a collaborative effort with contributions from all over the

world

R was initially written by Robert Gentleman and Ross Ihaka—also known as

"R & R" of the Statistics Department of the University of Auckland

R was inspired by the S environment

R can be extended (easily) via packages.

More about R

What R does and does not

ois not a database, but connects to DBMSsolanguage interpreter can be very slow, but allows to call own C/C++ code ono professional / commercial support

Data Types in R

numeric (integer, double, complex) character logical Data frame factor

Contributed Packages Currently, the CRAN package repository features 5034 available packages

Growing users of R

Text Mining: Basics

Text is Unstructured collections of words

Documents are basic units consisting of a sequence of tokens or terms

Terms are words or roots of words, semantic units or phrases which are the atoms of indexing

Repositories (databases) and corpora are collections of documents.

Corpus conceptual entity similar to a database for holding and managing text documents

Text mining involves computations to gain interesting information

Text Mining: Practical Applications

Spam filtering Business Intelligence, Marketing applications : predictive analytics Sentiment analysis Text IR, indexing Creating suggestion and recommendations (like amazon) Monitoring public opinions (for example in blogs or review sites) Customer service, email support Automatic labeling of documents in business libraries Fraud detection by investigating notification of claims Fighting cyberbullying or cybercrime in IM and IRC chatAnd many more

A List Text Mining Tools

Text Mining Packages in R

Corpora gsubfn kernlab KoNLP

koRpus `lda lsa maxent

movMF openNLP qdap RcmdrPlugin.temis

RKEA RTextTools Rweka skmeans

Snowball SnowballC tau textcat

Textir tm tm.plugin.dc tm.plugin.factiva

tm.plugin.mail topicmodels wordcloud

Wordnet zipfR

Text Mining Packages in Rplyr: Tools for splitting, applying and combining dataclass: Various functions for classificationtm: A framework for text mining applicationscorpora: Statistics and data sets for corpus frequency datasnowball: stemmersRweka: interface to Weka, a collection of ML algorithms for data mining taskswordnet: interface to WordNet using the Jawbone Java API to WordNetwordcloud: to make cloud of wordtextir: A suite of tools for text and sentiment miningtau: Text Analysis Utilitiestopicmodels: an interface to the C code for Latent Dirichlet Allocation (LDA) models and Correlated Topics Models (CTM)zipfR: Statistical models for word frequency distributions

Conceptual process in Text Mining

organize and structure the texts (into repository) convenient representation (preprocessing) Transform texts into structured formats (e.g. TDM)

The framework

different file formats and in different locations standardized interfaces to access the document (sources)

Metadata valuable insights into the document structure must be able to alleviate metadata usage

to efficiently work with the documents must provide tools and algorithm to perform common task (transformation) To extract patterns of interest (filtering)

Text document collections: CorpusConstructor:Corpus(object = ..., readerControl = list(reader = object@DefaultReader, language = "en_US", load = FALSE))

Example:>txt <- system.file("texts", "txt", package = "tm")>(ovid <- Corpus(DirSource(txt), readerControl = list(reader = readPlain, language = "la", load = TRUE))) A corpus with 5 text documents

Corpus: Meta Data>meta(ovid[[1]])Available meta data pairs are: Author : DateTimeStamp: 2013-11-19 18:54:04 Description : Heading : ID : ovid_1.txt Language : la Origin :

>ID(ovid[[1]]) [1] "ovid_1.txt“

Corpus: Document’s text>ovid[[1]] Si quis in hoc artem populo non novit amandi, hoc legat et lecto carmine doctus amet. arte citae veloque rates remoque moventur, arte leves currus: arte regendus amor. curribus Automedon lentisque erat aptus habenis, Tiphys in Haemonia puppe magister erat: me Venus artificem tenero praefecit Amori; Tiphys et Automedon dicar Amoris ego. ille quidem ferus est et qui mihi saepe repugnet: sed puer est, aetas mollis et apta regi. Phillyrides puerum cithara perfecit Achillem, atque animos placida contudit arte feros. qui totiens socios, totiens exterruit hostes, creditur annosum pertimuisse senem.

Corpus: Meta Data>c(ovid[1:2], ovid[3:4]) A corpus with 4 text documents

>length(ovid) 5

>summary(ovid) A corpus with 5 text documents The metadata consists of 2 tag-value pairs and a data frame Available tags are: create_date creator

Available variables in the data frame are: MetaID

Corpus: Meta Data

>CMetaData(ovid) $create_date [1] "2013-11-19 18:54:04 GMT" $creator [1] "“

>DMetaData(ovid) MetaID 1 0 2 0 3 0 4 0 5 0

Corpus: Transformations and Filters>getTransformations() [1] "as.PlainTextDocument" "removeNumbers" "removePunctuation" "removeWords" [5] "stemDocument" "stripWhitespace“

>tm_map(ovid, FUN = tolower)A corpus with 5 text documents

>getFilters() [1] "searchFullText" "sFilter" "tm_intersect"

>tm_filter(ovid, FUN = searchFullText, "Venus", doclevel = TRUE) A corpus with 1 text document

Text Preprocessing: import>txt <- system.file("texts", "crude", package = "tm") >(acq <- Corpus(DirSource(txt), readerControl = list(reader = readPlain, language = "la", load = TRUE))) A corpus with 50 text documents

>txt <- system.file("texts", "crude", package = "tm") >(crude <- Corpus(DirSource(txt), readerControl = list(reader = readPlain, language = "la", load = TRUE))) A corpus with 20 text documents

resulting in 50 articles of topic acq and 20 articles of topic crude

Preprocessing: stemming Morphological variants of a word (morphemes). Similar terms derived from

a common stem: engineer, engineered, engineeringuse, user, users, used, using

Stemming in Information Retrieval. Grouping words with a common stem together.

For example, a search on reads, also finds read, reading, and readable

Stemming consists of removing suffixes and conflating the resulting morphemes. Occasionally, prefixes are also removed.

Preprocessing: stemming Reduce terms to their “roots” automate(s), automatic, automation all reduced to automat.

for example compressed and compression are both accepted as equivalent to compress.

for exampl compress andcompress ar both acceptas equival to compress

Preprocessing: stemming

Typical rules in Stemming:

sses ssies iational atetional tion

Weight of word sensitive rules (m>1) EMENT →replacement → replaccement → cement

Preprocessing: stemming

help recall for some queries but harm precision on others Fine distinctions may be lost through stemming.

Preprocessing: stemming>acq[[10]] Gulf Applied Technologies Inc said it sold its subsidiaries engaged in pipeline and terminal operations for 12.2 mln dlrs. The company said the sale is subject to certain post closing adjustments, which it did not explain. Reuter

>stemDocument(acq[[10]]) Gulf Appli Technolog Inc said it sold it subsidiari engag in pipelin and terminal oper for 12.2 mln dlrs. The compani said the sale is subject to certain post clos adjustments, which it did not explain. Reuter

>tm_map(acq, stemDocument) A corpus with 50 text documents

Preprocessing: Whitespace elimination & lower case conversion

>stripWhitespace(acq[[10]]) Gulf Applied Technologies Inc said it sold its subsidiaries engaged in pipeline and terminal operations for 12.2 mln dlrs. The company said the sale is subject to certain post closing adjustments, which it did not explain. Reuter

>tolower(acq[[10]]) gulf applied technologies inc said it sold its subsidiaries engaged in pipeline and terminal operations for 12.2 mln dlrs. the company said the sale is subject to certain post closing adjustments, which it did not explain. reuter

Preprocessing: Stopword removal

Very common words, such as of, and, the, are rarely of use in information retrieval.A long stop list saves space in indexes, speeds processing, and eliminates many false hits.However, common words are sometimes significant in information retrieval, which is an argument for a short stop list.

(Consider the query, "To be or not to be?")

Preprocessing: Stopword removal

Include the most common words in the English language (perhaps 50 to 250 words).

Do not include words that might be important for retrieval (Among the 200 most frequently occurring words in general literature in English are time, war, home, life, water, and world).

In addition, include words that are very common in context (e.g., computer, information, system in a set of computing documents).

Preprocessing: Stopword removal

about above accordingacross actually adj after afterwardsagain against all almost alone along already

also although always among amongst an another any anyhow anyone anything anywhere are aren't around at be became because become becomes becoming been before beforehand begin beginning behind being below beside besides between beyond billion both but by can can't cannot caption co could couldn't did didn't do does doesn't don't down during each eg eight eighty either else elsewhere end ending enough etc even ever every everyone everything

Preprocessing: Stopword removal

How many words should be in the stop list?• Long list lowers recall

Which words should be in list?• Some common words may have retrieval importance:

-- war, home, life, water, world

• In certain domains, some words are very common:-- computer, program, source, machine, language

Preprocessing: Stopword removal

>mystopwords <- c("and", "for", "in", "is", "it", "not", "the", "to")

>removeWords(acq[[10]], mystopwords) Gulf Applied Technologies Inc said sold its subsidiaries engaged pipeline terminal operations 12.2 mln dlrs. The company said sale subject certain post closing adjustments, which did explain. Reuter

>tm_map(acq, removeWords, mystopwords) A corpus with 50 text documents

Preprocessing: Synonyms

> library("wordnet")

synonyms("company")[1] "caller" "companionship" "company" "fellowship"[5] "party" "ship’s company" "society" "troupe“

replaceWords(acq[[10]], synonyms(dict, "company"), by = "company")

Tm_map(acq, replaceWords, synonyms(dict, "company"), by = "company")

Preprocessing: Part of speech tagging>library("NLP","openNLP")s <- as.String(acq[[10]])## Need sentence and word token annotations.sent_token_annotator <- Maxent_Sent_Token_Annotator()word_token_annotator <- Maxent_Word_Token_Annotator()a2 <- annotate(s, list(sent_token_annotator, word_token_annotator))pos_tag_annotator <- Maxent_POS_Tag_Annotator()#pos_tag_annotatora3 <- annotate(s, pos_tag_annotator, a2)a3w <- subset(a3, type == "word")tags <- sapply(a3w$features, "[[", "POS")sprintf("%s/%s", s[a3w], tags)

Preprocessing: Part of speech tagging"Gulf/NNP" "Applied/NNP" "Technologies/NNP" "Inc/NNP" "said/VBD" "it/PRP" "sold/VBD" "its/PRP$" "subsidiaries/NNS" "engaged/VBN" "in/IN" "pipeline/NN" "and/CC" "terminal/NN" "operations/NNS" "for/IN" "12.2/CD" "mln/NN" "dlrs/NNS" "./." "The/DT" "company/NN" "said/VBD" "the/DT" "sale/NN" "is/VBZ" "subject/JJ" "to/TO" "certain/JJ" "post/NN" "closing/NN" "adjustments/NNS" ",/," "which/WDT" "it/PRP" "did/VBD" "not/RB" "explain/VB" "./." "Reuter/NNP“more

Preprocessing

R Demo

Classification using KNN

K-Nearest Neighbor algorithm: Most basic instance-based method Data are represented in a vector space Supervised learning

, V is the finite set {v1,......,vn}

the k-NN returns the most common value among the k training examples nearest to xq.

KNN Feature space

KNN Training algorithm

For each training example <x,f(x)> add the example to the listClassification algorithm

Given a query instance xq to be classifiedLet x1,..,xk k instances which are nearest to xq

Where (a,b)=1 if a=b, else (a,b)= 0 (Kronecker function)

Classification using KNN : Example

Two classes: Red and Blue

Green is Unknown

With K=3, classification is RedWith k=4, classification is Blue

How to determine the good value for k?

Determined experimentally Start with k=1 and use a test set to validate the error rate of the classifier Repeat with k=k+2 Choose the value of k for which the error rate is minimum

Note: k should be odd number to avoid ties

KNN for speech classificationDatasets: Size: 40 instancesBarak Obama 20 speeches Mitt Romney 20 speeches

Training datasets: 70% (28)Test datasets: 30% (12)

Accuracy: on average more than 90%

Speech Classification Implementation in R#initialize the R environmentlibs<-c("tm","plyr","class")lapply(libs,require,character.only=TRUE)

#Set parameters / source directorydir.names<-c("obama","romney")path<-"E:/Ashraf/speeches"

#clean text / preprocessingcleanCorpus<-function(corpus){ corpus.tmp<-tm_map(corpus,removePunctuation) corpus.tmp<-tm_map(corpus.tmp,stripWhitespace) corpus.tmp<-tm_map(corpus.tmp,tolower) corpus.tmp<-tm_map(corpus.tmp,removeWords,stopwords("english")) return (corpus.tmp)}

Speech Classification Implementation in R#build term document matrixgenerateTDM<-function(dir.name,dir.path){ s.dir<-sprintf("%s/%s",dir.path,dir.name) s.cor<-Corpus(DirSource(directory=s.dir,encoding="ANSI")) s.cor.cl<-cleanCorpus(s.cor) s.tdm<-TermDocumentMatrix(s.cor.cl) s.tdm<-removeSparseTerms(s.tdm,0.7) result<-list(name=dir.name,tdm=s.tdm)}

tdm<-lapply(dir.names,generateTDM,dir.path=path)

Speech Classification Implementation in R#attach candidate name to each row of TDMbindCandidateToTDM<-function(tdm){ s.mat<-t(data.matrix(tdm[["tdm"]])) s.df<-as.data.frame(s.mat,StringAsFactors=FALSE) s.df<-cbind(s.df,rep(tdm[["name"]],nrow(s.df))) colnames(s.df)[ncol(s.df)]<-"targetcandidate" return (s.df)}

candTDM<-lapply(tdm,bindCandidateToTDM)

Speech Classification Implementation in R#stack the TDMs together (for both Obama and Romnie)tdm.stack<-do.call(rbind.fill,candTDM)tdm.stack[is.na(tdm.stack)]<-0

#hold-out / splitting training and test data setstrain.idx<-sample(nrow(tdm.stack),ceiling(nrow(tdm.stack)*0.7))test.idx<-(1:nrow(tdm.stack))[-train.idx])

Speech Classification Implementation in R#model KNNtdm.cand<-tdm.stack[,"targetcandidate"]tdm.stack.nl<-tdm.stack[,!colnames(tdm.stack)%in%"targetcandidate"]

knn.pred<-knn(tdm.stack.nl[train.idx,],tdm.stack.nl[test.idx,],tdm.cand[train.idx])

#accuracy of the predictionconf.mat<-table('Predictions'=knn.pred,Actual=tdm.cand[test.idx])(accuracy<-(sum(diag(conf.mat))/length(test.idx))*100)

#show resultshow(conf.mat)show(accuracy)

Speech Classification Implementation in R

Show R Demo

References

1. Text Mining Infrastructure in R, Ingo Feinerer, Kurt Hornik, David Meyer, Vol. 25, Issue 5, Mar 2008, Journal of Statistical Software.

2. http://mittromneycentral.com/speeches/3. http://obamaspeeches.com/4. http://cran.r-project.org/