ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA...
-
Upload
tylor-gladding -
Category
Documents
-
view
258 -
download
4
Transcript of ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA...
![Page 1: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/1.jpg)
ROC curves
Data Mining Lab 5
![Page 2: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/2.jpg)
Lab outline
• Remind what ROC curve is
• Generate ROC curves using WEKA
• Some usage of ROC curves
![Page 3: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/3.jpg)
Point in ROC space
TRUE CLASS
YES NO
PREDICTED CLASS
YES TP FP
NO FN TN
Total: P N
FP rate: FP/N TP rate: TP/P (recall)
FN rate: FN/N TN rate: TN/P
Classifier accuracy: (TP+TN)/(P+N)
FP rate
TP rate
Shows how good is classifier in discriminating positive instances from negative ones
![Page 4: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/4.jpg)
ROC curve of a probabilistic classifier
Outlook Temp Windy P(Y|E) Real class
overcast mild yes 0.95 YES
rainy mild no 0.80 YES
rainy cool yes 0.60 NO
sunny mild no 0.45 YES
sunny cool no 0.40 NO
sunny hot no 0.35 NO
sunny hot yes 0.25 NO
Naïve Bayes, for example, outputs the probability of an instance in a testing set to be classified as YES
![Page 5: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/5.jpg)
ROC curve of a probabilistic classifier
Outlook Temp Windy P(Y|E) Real class
overcast mild yes 0.95 YES
rainy mild no 0.80 YES
rainy cool yes 0.60 NO
sunny mild no 0.45 YES
sunny cool no 0.40 NO
sunny hot no 0.35 NO
sunny hot yes 0.25 NO
In a general case, we classify an instance as YES if the probability is more than 50%
Classified as YES
Classified as NO
Operating threshold
![Page 6: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/6.jpg)
ROC curve of a probabilistic classifier
Outlook Temp
Windy P(Y|E) Predicted class
Real class
overcast mild yes 0.95 YES YES
rainy mild no 0.80 YES YES
rainy cool yes 0.60 YES NO
sunny mild no 0.45 NO YES
sunny cool no 0.40 NO NO
sunny hot no 0.35 NO NO
sunny hot yes 0.25 NO NO
We compute the confusion matrix
TRUE CLASS
YES NO
PREDICTED CLASS
YES 2 (TP) 1 (FP)
NO 1 (FN) 3 (TN)
Total: 3 (P) 4 (N)
And the TP and FP rates:
TP rate: TP/P=2/3≈0.7
FP rate: FP/N=1/4=0.25
![Page 7: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/7.jpg)
ROC curve of a probabilistic classifier
Outlook Temp
Windy P(Y|E) Predicted class
Real class
overcast mild yes 0.95 YES YES
rainy mild no 0.80 YES YES
rainy cool yes 0.60 YES NO
sunny mild no 0.45 NO YES
sunny cool no 0.40 NO NO
sunny hot no 0.35 NO NO
sunny hot yes 0.25 NO NO
This corresponds to point A in a ROC space
FP rate: FP/N=1/4=0.25
TP rate: TP/P=2/3≈0.7
FP rate
TP rate
1
1
A
![Page 8: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/8.jpg)
ROC curve of a probabilistic classifier
Outlook Temp
Windy P(Y|E) Predicted class
Real class
overcast mild yes 0.95 YES YES
rainy mild no 0.80 YES YES
rainy cool yes 0.60 YES NO
sunny mild no 0.45 NO YES
sunny cool no 0.40 NO NO
sunny hot no 0.35 NO NO
sunny hot yes 0.25 NO NO
For different threshold values we get different points in a ROC space
FP rate: FP/N=0/4=0
TP rate: TP/P=1/3≈0.3
FP rate
TP rate
1
1
![Page 9: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/9.jpg)
ROC curve of a probabilistic classifier
Outlook Temp
Windy P(Y|E) Predicted class
Real class
overcast mild yes 0.95 YES YES
rainy mild no 0.80 YES YES
rainy cool yes 0.60 YES NO
sunny mild no 0.45 NO YES
sunny cool no 0.40 NO NO
sunny hot no 0.35 NO NO
sunny hot yes 0.25 NO NO
For different threshold values we get different points in a ROC space
FP rate: FP/N=0/4=0
TP rate: TP/P=2/3≈0.7
FP rate
TP rate
1
1
![Page 10: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/10.jpg)
ROC curve of a probabilistic classifier
Outlook Temp
Windy P(Y|E) Predicted class
Real class
overcast mild yes 0.95 YES YES
rainy mild no 0.80 YES YES
rainy cool yes 0.60 YES NO
sunny mild no 0.45 NO YES
sunny cool no 0.40 NO NO
sunny hot no 0.35 NO NO
sunny hot yes 0.25 NO NO
For different threshold values we get different points in a ROC space
FP rate: FP/N=1/4=0.25
TP rate: TP/P=2/3≈0.7
FP rate
TP rate
1
1
![Page 11: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/11.jpg)
ROC curve of a probabilistic classifier
Outlook Temp
Windy P(Y|E) Predicted class
Real class
overcast mild yes 0.95 YES YES
rainy mild no 0.80 YES YES
rainy cool yes 0.60 YES NO
sunny mild no 0.45 YES YES
sunny cool no 0.40 NO NO
sunny hot no 0.35 NO NO
sunny hot yes 0.25 NO NO
For different threshold values we get different points in a ROC space
FP rate: FP/N=1/4=0.25
TP rate: TP/P=3/3=1.0, etc…
FP rate
TP rate
1
1
![Page 12: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/12.jpg)
ROC curve of a probabilistic classifier
At the end we get the ROC curve for Naïve Bayes classifier
FP rate
TP rate
1
1
![Page 13: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/13.jpg)
ROC curve of a probabilistic classifier vs discrete classifier
ROC curve for Naïve Bayes classifier
(probabilistic)
FP rate
TP rate
1
1
FP rate
TP rate
1
1
ROC curve for Decision Tree classifier
(discrete)
![Page 14: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/14.jpg)
Lab outline
• Remind what ROC curve is
• Generate ROC curves using WEKA
• Some usage of ROC curves
![Page 15: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/15.jpg)
Preparation Step 1. Increase Java heap size
Step 2. Download input data file
into your home directory
![Page 16: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/16.jpg)
Comparing classifiers. Knowledge flow
![Page 17: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/17.jpg)
Knowledge flow tabs
DATA SOURCES
FILTERS
CLASSIFIERS
EVALUATION
VISUALIZATION
![Page 18: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/18.jpg)
Loading the data
Click
![Page 19: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/19.jpg)
Loading the data
Select file adult_income.arff
![Page 20: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/20.jpg)
Data file adult_income.arff@relation adults
1. @attribute age numeric2. @attribute workclass {Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-
gov, State-gov, Without-pay, Never-worked}3. @attribute education real4. @attribute marital_status {Married-civ-spouse, Divorced, Never-married, Separated,
Widowed, Married-spouse-absent, Married-AF-spouse}5. @attribute occupation {Tech-support, Craft-repair, Other-service, Sales, Exec-
managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces}
6. @attribute sex {Male, Female}7. @attribute native_country {United-States, Cambodia, England, Puerto-Rico, Canada,
Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands}
8. @attribute class {>50K, <=50K}
Data from US census
![Page 21: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/21.jpg)
Attributes of interest: age, education,
class (income >50 K: YES,NO)1. @attribute age numeric3. @attribute education reallast @attribute class {>50K, <=50K}
Numeric codes for education levels
Doctorate,16 Prof-school, 15Masters, 14Bachelors, 13Assoc-acdm, 12Assoc-voc, 11Some-college, 10HS-grad, 911th, -712th, 810th, 69th, 57th-8th, 45th-6th,31st-4th, 2Preschool, 1
We remove all other attributes and leave only attributes 1,3, last – for simplicity
We build a classifier, which predicts income based on age and education level.
![Page 22: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/22.jpg)
Removing attributes
![Page 23: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/23.jpg)
Removing attributes
![Page 24: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/24.jpg)
Removing attributes
![Page 25: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/25.jpg)
Removing attributes
Type
Means remove all except attributes 1,3,last
![Page 26: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/26.jpg)
Visualize data
![Page 27: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/27.jpg)
Visualize data
![Page 28: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/28.jpg)
Connect the flow
![Page 29: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/29.jpg)
Connect the flow: from data loader to attribute remover
![Page 30: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/30.jpg)
Connect the flow: from attribute remover to summarizer
![Page 31: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/31.jpg)
Start data flow
![Page 32: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/32.jpg)
Visualize the data
![Page 33: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/33.jpg)
Visualize the data
>50K
<=50K
![Page 34: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/34.jpg)
Assigning the class
![Page 35: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/35.jpg)
Configuring class assigner
![Page 36: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/36.jpg)
Subdivision of the dataset into “learning” and “test” set
![Page 37: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/37.jpg)
Subdivision of the dataset into “learning” and “test” set
We want to build our prediction model on the 70% of the whole dataset,
and compute theROC curve on the remaining.
So, we set the TRAINTEST SPLIT MAKER (EVALUATION) inthe diagram and configure its parameters.
![Page 38: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/38.jpg)
Choosing discrete classifier – decision tree
![Page 39: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/39.jpg)
Connecting classifier to the data
We set J48 component in the diagram,
we connect twice the TRAIN TEST SPLIT MAKER to this new component: twice because wemust use together the training and the test set which are produced by the same component.
![Page 40: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/40.jpg)
Adding visualizer to see the classification results
![Page 41: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/41.jpg)
Perform classification
![Page 42: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/42.jpg)
Show classification results (decision tree)
Assoc-acdm
Masters
![Page 43: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/43.jpg)
Classifier evaluation
![Page 44: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/44.jpg)
Connecting classifier to the evaluator
![Page 45: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/45.jpg)
Selecting performance model: chart
![Page 46: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/46.jpg)
Running the model
![Page 47: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/47.jpg)
View ROC curve
![Page 48: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/48.jpg)
Adding Naïve Bayes classifier
![Page 49: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/49.jpg)
Adding separate performance evaluator for Naïve Bayes classifier
![Page 50: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/50.jpg)
Connecting second performance evaluator to the same Model Performance Chart
![Page 51: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/51.jpg)
Run both classifiers
![Page 52: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/52.jpg)
View ROC curves for both classifiers
![Page 53: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/53.jpg)
Lab outline
• Remind what ROC curve is
• Generate ROC curves using WEKA
• Some usage of ROC curves
![Page 54: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/54.jpg)
Compare classifiers using their ROC curves
![Page 55: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/55.jpg)
How good is the classifier
The area under the ROC curve shows the quality of a classifier – not accuracy, but the ability to separate between positive and negative instances.
What classifier is better?
![Page 56: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/56.jpg)
Choosing the Operating Point
• Usually a classifier is used at a particular sensitivity, or at a particular threshold. The ROC curve can be used to choose the best operating point. The best operating point might be chosen so that the classifier gives the best trade off between the costs of failing to detect positives against the costs of raising false alarms. These costs need not be equal, however this is a common assumption.
• The best place to operate the classifier is the point on its ROC which lies on a 45 degree line closest to the north-west corner (0,1) of the ROC plot.
DBL Click
![Page 57: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/57.jpg)
Cost sensitive operating points
A
Is this threshold good :
for cancer detection?
for targeting potential customers?
![Page 58: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/58.jpg)
Cost sensitive operating points
BIs this threshold good :
for cancer detection?
for targeting potential customers?
![Page 59: ROC curves Data Mining Lab 5. Lab outline Remind what ROC curve is Generate ROC curves using WEKA Some usage of ROC curves.](https://reader036.fdocuments.net/reader036/viewer/2022081421/5517fddf550346d5568b5097/html5/thumbnails/59.jpg)
Conclusions
• WEKA is a powerful datamining tool, but is not very easy to use
• There are other open source data mining tools, which are easier to use:– Orange:
• http://www.ailab.si/orange – Tanagra:
• http://eric.univ-lyon2.fr/~ricco/tanagra/en/tanagra.html