SPACE Redefining our surroundings. Questioning definitions of shelter Moshe Moshe Safdie.
Text Categorization Moshe Koppel Lecture 1: Introduction
25
Text Categorization Moshe Koppel Lecture 1: Introduction Slides based on Manning, Raghavan and Schutze and odds and ends from here and there
-
Upload
nathaniel-cervantes -
Category
Documents
-
view
46 -
download
3
description
Text Categorization Moshe Koppel Lecture 1: Introduction. Slides based on Manning, Raghavan and Schutze and odds and ends from here and there. Text Classification. Text classification (text c ategorization ): assign documents to one or more predefined categories - PowerPoint PPT Presentation
Transcript of Text Categorization Moshe Koppel Lecture 1: Introduction
Text CategorizationMoshe Koppel
Lecture 1: Introduction
Slides based on Manning, Raghavan and Schutze and odds and ends from here and there
Text Classification
• Text classification (text categorization):assign documents to one or more predefined categories
classes Documents ? class1 class2 . . . classn
Illustration of Text Classification
Science
Sport
Art
EXAMPLES OF TEXT CATEGORIZATION
• LABELS=TOPICS– “finance” / “sports” / “asia”
• LABELS=AUTHOR– “Shakespeare” / “Marlowe” / “Ben Jonson”– The Federalist papers
• LABELS=OPINION– “like” / “hate” / “neutral “
• LABELS=SPAM?– “spam” / “not spam”
Text Classification Framework
Documents PreprocessingFeatures/Indexing
Feature filtering
Applyingclassificationalgorithms
Performancemeasure
Preprocessing
• Preprocessing:
transform documents into a suitable representation for classification task
– Remove HTML or other tags
– Remove stopwords
– Perform word stemming (Remove suffix)
Features+Indexing
• Feature types (task dependent)
• Measure
Feature types
Most crucial decision you’ll make!1. Topic
• Words, phrases, ?
2. Author• Stylistic features
3. Sentiment• Adjectives, ?
4. Spam• Specialized vocabulary
Indexing
• Indexing by different weighing schemes:– Boolean weighting– word frequency weighting– tf*idf weighting– entropy weighting
Feature Selection
• Feature selection:
remove non-informative terms from documents
=>improve classification effectiveness
=>reduce computational complexity
Evaluation measures
• Precision wrt ci
• Recall wrt ci
ii
ii
FPTP
TP
ii
ii
FNTP
TP
TPiFPi FNi
TNi
Classified Ci Test Class Ci
Combined effectiveness measures
• a classifier should be evaluated by means of a measure which combines recall and precision (why?)
• some combined measures:– F1 measure
– the breakeven point
F1 measure
• F1 measure is defined as:
• for the trivial acceptor, 0 and = 1, F1 0
2
1F
Breakeven point
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.2 0.4 0.6 0.8 1Precision
Recall
breakeven point is the value where precision equals recall
Multiclass Problem: Micro- vs. Macro-Averaging
• If we have more than one class, how do we combine multiple performance measures into one quantity?
• Macroaveraging: Compute performance for each class, then average.
• Microaveraging: Collect decisions for all classes, compute contingency table, evaluate.
Experiments
• Topic-based categorization• Burst of experiments around 1998• Content features ~ words• Experiments focused on algorithms• Some focused on feature filtering (next lecture)• Standard corpus: Reuters
Reuters-21578: Typical document<REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="12981" NEWID="798">
<DATE> 2-MAR-1987 16:51:43.42</DATE>
<TOPICS><D>livestock</D><D>hog</D></TOPICS>
<TITLE>AMERICAN PORK CONGRESS KICKS OFF TOMORROW</TITLE>
<DATELINE> CHICAGO, March 2 - </DATELINE><BODY>The American Pork Congress kicks off
tomorrow, March 3, in Indianapolis with 160 of the nations pork producers from 44 member states determining industry positions on a number of issues, according to the National Pork Producers Council, NPPC.
Delegates to the three day Congress will be considering 26 resolutions concerning various issues, including the future direction of farm policy and the tax law as it applies to the agriculture sector. The delegates will also debate whether to endorse concepts of a national PRV (pseudorabies virus) control and eradication program, the NPPC said.
A large trade show, in conjunction with the congress, will feature the latest in technology in all areas of the industry, the NPPC added. Reuter
</BODY></TEXT></REUTERS>
• Most (over)used data set (c. 1998)
• 21578 documents
• Average document length: 200 words
• 9603 training, 3299 test articles (ModApte split)
• 118 categories
• article can be in > 1 category (average: 1.24)
• only about 10 out of 118 categories are large
Common categories(#train, #test)
Reuters 21578
• Earn (2877, 1087) • Acquisitions (1650, 179)• Money-fx (538, 179)• Grain (433, 149)• Crude (389, 189)
• Trade (369,119)• Interest (347, 131)• Ship (197, 89)• Wheat (212, 71)• Corn (182, 56)
First Experiment: Yang and Liu
• Features: stemmed words (stop words removed)
• Indexing: frequency (?)
• Feature filtering: top infogain words (1000 to 10000)
• Evaluation: macro- and micro-averaged F1
Results: Yang&Liu
Second Experiment: Dumais et al
• Features: non-rare words
• Indexing: binary
• Feature filtering: top infogain words (30 per category)
• Evaluation: macro-averaged break-even
Results: Dumais et al. Breakeven
Rocchio NBayes C4.5 LinearSVMearn 92.9% 95.9% 97.8% 98.2%acq 64.7% 87.8% 89.7% 92.8%money-fx 46.7% 56.6% 66.2% 74.0%grain 67.5% 78.8% 85.0% 92.4%crude 70.1% 79.5% 85.0% 88.3%trade 65.1% 63.9% 72.5% 73.5%interest 63.4% 64.9% 67.1% 76.3%ship 49.2% 85.4% 74.2% 78.0%wheat 68.9% 69.7% 92.5% 89.7%corn 48.2% 65.3% 91.8% 91.1%
Avg Top 10 64.6% 81.5% 88.4% 91.4%Avg All Cat 61.7% 75.2% na 86.4%
Observations: Dumais et al
• Features: words + bigrams
No improvement!
• Indexing: frequency instead of binary
No improvement!
Third Experiment: Joachims
• Features: stemmed unigrams (stop words removed)
• Indexing: tf*idf
• Feature filtering: 1000 top infogain words
• Evaluation: micro-averaged break-even
Results: Joachims
