1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.
-
date post
20-Dec-2015 -
Category
Documents
-
view
220 -
download
2
Transcript of 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.
![Page 1: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/1.jpg)
1
I256: Applied Natural Language Processing
Marti HearstSept 27, 2006
![Page 2: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/2.jpg)
2
Evaluation Measures
![Page 3: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/3.jpg)
3
Evaluation Measures
Precision: Proportion of those you labeled X that the gold standard thinks really is X #correctly labeled by alg/ all labels assigned by alg #True Positive / (#True Positive + #False Positive)
Recall:Proportion of those items that are labeled X in the gold standard that you actually label X#correctly labeled by alg / all possible correct labels#True Positive / (#True Positive + # False Negative)
![Page 4: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/4.jpg)
4
F-measure
Can “cheat” with precision scores by labeling (almost) nothing with X.Can “cheat” on recall by labeling everything with X.The better you do on precision, the worse on recall, and vice versaThe F-measure is a balance between the two.
2*precision*recall / (recall+precision)
![Page 5: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/5.jpg)
5
Evaluation Measures
Accuracy:Proportion that you got right (#True Positive + #True Negative) / N
N = TP + TN + FP + FN
Error:(#False Positive + #False Negative)/N
![Page 6: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/6.jpg)
6
Prec/Recall vs. Accuracy/Error
When to use Precision/Recall?Useful when there are only a few positives and many many negativesAlso good for ranked ordering
– Search results ranking
When to use Accuracy/ErrorWhen every item has to be judged, and it’s important that every item be correct.Error is better when the differences between algorithms are very small; let’s you focus on small improvements.
– Speech recognition
![Page 7: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/7.jpg)
7
Evaluating Partial Parsing
How do we evaluate it?
![Page 8: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/8.jpg)
8
Evaluating Partial Parsing
![Page 9: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/9.jpg)
9
Testing our Simple Fule
Let’s see where we missed:
![Page 10: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/10.jpg)
10
Update rules; Evaluate Again
![Page 11: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/11.jpg)
11
Evaluate on More Examples
![Page 12: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/12.jpg)
12
Incorrect vs. MissedAdd code to print out which were incorrect
![Page 13: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/13.jpg)
13
Missed vs. Incorrect
![Page 14: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/14.jpg)
14
What is a good Chunking Baseline?
![Page 15: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/15.jpg)
15
![Page 16: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/16.jpg)
16
The Tree Data Structure
![Page 17: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/17.jpg)
17
Baseline Code (continued)
![Page 18: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/18.jpg)
18
Evaluating the Baseline
![Page 19: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/19.jpg)
19
Cascaded Chunking
![Page 20: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/20.jpg)
20
![Page 21: 1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.](https://reader035.fdocuments.net/reader035/viewer/2022062421/56649d4c5503460f94a2ad9e/html5/thumbnails/21.jpg)
21
Next Time
Summarization