Comprehensive Introduction to the Evaluation of Neural Networks and other Computational Intelligence...
-
Upload
tyler-nash -
Category
Documents
-
view
218 -
download
0
Transcript of Comprehensive Introduction to the Evaluation of Neural Networks and other Computational Intelligence...
Comprehensive Introduction to the Evaluation of Neural Networks and other Computational Intelligence Decision Functions: Receiver Operating Characteristic, Jackknife,
Bootstrap and other Statistical Methodologies
David G. Brown and Frank Samuelson
Center for Devices and Radiological Health, FDA
6 July 2014
Course OutlineI. Performance measures for Computational Intelligence (CI) observers
1. Accuracy
2. Prevalence dependent measures
3. Prevalence independent measures
4. Maximization of performance: Utility analysis/Cost functions
II. Receiver Operating Characteristic (ROC) analysis1. Sensitivity and specificity
2. Construction of the ROC curve
3. Area under the ROC curve (AUC)
III. Error analysis for CI observers1. Sources of error
2. Parametric methods
3. Nonparametric methods
4. Standard deviations and confidence intervals
IV. Boot strap methods1. Theoretical foundation
2. Practical use
V. References
What’s the problem?
• Emphasis on algorithm innovation to exclusion of performance assessment
• Use of subjective measures of performance – “beauty contest”
• Use of “accuracy” as a measure of success• Lack of error bars—My CIO is .01 better than
yours (+/- ?)• Flawed methodology—training and testing on
same data • Lack of appreciation for the many different
sources of error that can be taken into account
Original image
Lena. Courtesy of the Signal and Image Processing Institute at the University of Southern California.
CI improved image
Baboon. Courtesy of the Signal and Image Processing Institute at the University of Southern California.
Panel of expertsfunnymonkeysite.com
I. Performance measures for computational intelligence (CI) observers
• Task based: (binary) discrimination task– Two populations involved: “normal” and “abnormal,”
• Accuracy – Intuitive but incomplete– Different consequences for success or failure for each
population
• Some measures depend on the prevalence (Pr) some do not, Pr = – Accuracy, positive predictive value, negative predictive value– Sensitivity, specificity, ROC, AUC
• True optimization of performance requires knowledge of cost functions or utilities for successes and failures in both populations
populationin subjects all ofnumber Total
populationin abnormals ofNumber
How to make a CIO with >99% accuracy
• Medical problem: Screening mammography (“screening” means testing in an asymptomatic population)
• Prevalence of breast cancer in the screening population Pr = 0.5 %
• My CIO always says “normal”• Accuracy (Acc) is 99.5% (accuracy of accepted
present-day systems ~75%)• Accuracy in a diagnostic setting (Pr~20%) is
80% -- Acc=1-Pr (for my CIO)
CIO operates on two different populations
Normal cases p(t|0)
Abnormal casesp(t|1)
Threshold t = T
t-axis
Must consider effects on normal and abnormal populations separately
• CIO output t• p(t|0) probability distribution of t for the population of normals• p(t|1) probability distribution of t for the population of abnormals• Threshold T. Everything to the right of T called abnormal, and
everything to the left of T called normal• Area of p(t|0) to left of T is the true negative fraction (TNF =
specificity) and to the right the false positive fraction (FPF = type 1 error). TNF + FPF = 1
• Area of p(t|1) to left of T is the false negative fraction (FNF = type 2 error) and to the right is the true positive fraction (TPF = sensitivity) FNF + TPF = 1
• TNF, FPF, FNF, TPF all are prevalence independent, since each is some fraction of one of our two probability distributions
• {Accuracy = Pr x TPF + (1-Pr) x TNF}
Normalcases
Abnormalcases
Threshold T
TPF (.95)
TNF (.5)
t-axis
t-axis
FNF (.05)
FPF (.5)
Prevalence dependent measures
• Accuracy (Acc) Acc = Pr x TPF + (1-Pr) x TNF
• Positive predictive value (PPV): fraction of positives that are true positives PPV = TPF x Pr / (TPF x Pr + FPF x (1-Pr))
• Negative predictive value (NPV): fraction of negatives that are true negativesNPV = TNF x (1-Pr) / (TNF x (1-Pr) + FNF x Pr)
• Using the mammography screening Pr and previous TPF, TNF, FNF, FPF values: Pr = .05, TPF = .95, TNF = 0.5, FNF=.05, FPF=0.5 Acc = .05x.95+.95x.5 = .52 PPV = .95x.05/(.95x.05+.5x.95) = .10NPV = .5x.95/(.5x.95+.05x.05) = .997
Prevalence dependent measures
• Accuracy (Acc) Acc = Pr x TPF + (1-Pr) x TNF
• Positive predictive value (PPV): fraction of positives that are true positives PPV = TPF x Pr / (TPF x Pr + FPF x (1-Pr))
• Negative predictive value (NPV): fraction of negatives that are true negativesNPV = TNF x (1-Pr) / (TNF x (1-Pr) + FNF x Pr)
• Using the mammography screening Pr and previous TPF, TNF, FNF, FPF values: Pr = .005, TPF = .95, TNF = 0.5, FNF=.05, FPF=0.5 Acc = .005x.95+.995x.5 = .50 PPV = .95x.005/(.95x.005+.5x.995) = .01NPV = .5x.995/(.5x.995+.05x.005) = .995
Acc, PPV, NPV as functions of prevalence(screening mammography)
• TPF=.95• FNF=.05• TNF=0.5• FPF=0.5
Acc = NPV as function of prevalence(forced “normal” response CIO)
Prevalence independent measures
• Sensitivity = TPF
• Specificity = TNF (1-FPF)
• Receiver Operating Characteristic (ROC) = TPF as a function of FPF (Sensitivity as a function of 1 – Specificity)
• Area under the ROC curve (AUC)
= Sensitivity averaged over all values of Specificity
17
Threshold
TP
F,
sens
itivi
tyFPF, 1-specificity
Entire ROC curve
Normal / Class 0subjects
Abnormal / Class 1subjects
ROC slope
Empirical ROC data for mammography screening in the US
0 .0
0 .0
0 .0
0 .0
0 .1
0 .1
0 .1
0 .1
0 .2
0 .2
0 .2
0 .2
0 .3
0 .3
0 .3
0 .3
0 .4
0 .4
0 .4
0 .4
0 .5
0 .5
0 .5 0 .5
0 .6
0 .6
0 .6
0 .6
0 .7
0 .7
0 .7
0 .7
0 .8
0 .8
0 .8
0 .8
0 .9
0 .9
0 .9
0 .9
1 .0
1 .01 .0
1 .0
F a lse P o sitiv e F ractio n
Tru e N ega tiv e F ractio n
Tru
e P
osit
ive
Fra
ctio
n
Fal
se N
egat
ive
Fra
ctio
n
Craig Beam et al.
Maximization of performance• Need to know utilities or costs of each type of decision outcome –
but these are very hard to estimate accurately. You don’t just maximize accuracy.
• Need prevalence• For mammography example
– TPF: prolongation of life minus treatment cost– FPF: diagnostic work-up cost, anxiety– TNF: peace of mind– FNF: delay in treatment => shortened life
• Hypothetical assignment of utilities for some decision threshold T:– UtilityT= U(TPF) x TPF x Pr + U(FPF) x FPF x (1-Pr)
+ U(TNF) x TNF x (1-Pr) + U(FNF) x FNF x Pr– U(TPF) = 100, U(FPF) = -10, U(TNF) = 4, U(FNF) = -20– UtilityT= 100 x .95 x .05 – 10 x .50 x .95
+ 4 x .50 x .95 – 20 x .05 x .05 = 1.85• Now if we only knew how to trade off TPF versus FPF, we could
optimize (?) medical performance.
Utility maximization(mammography example)
Choice of ROC operating point through utility analysis—screening mammography
Utility maximization(mammography example)
Utility maximization calculation
UTPFTPF+UFNFFNF)PR+(UTNFTNF+UFPFFPF)(1-PR)
=(UTPFTPF+UFNF(1-TPF))PR+(UTNF(1-FPF)+UFPFFPF)(1-PR)
d/dFPF=(UFPF-UTNF)(1-PR)+(UTPF-UFNF)PRdTPF/dFPF
=0 dTPF/dFPF=(UTNF-UFPF)(1-PR)/(UTPF-UFNF)PR
PR=.005 dTPF/dFPF = 23.
PR=.05 dTPF/dFPF = 2.2
(UTPF=100, UFNF=-20, UTNF=4, UFPF=-20)
Threshold
Abnormalcases
TPF,
sensi
tivit
yFPF, 1-specificity
Entire ROC curve
Normalcases
ROC slope
Estimators
• TPF, FPF, TNF, FNF, Accuracy, the ROC curve, and AUC are all fractions or probabilities.
• Normally we have a finite sample of subjects on which to test our CIO. From this finite sample we try to estimate the above fractions– These estimates will vary depending upon the
sample selected (statistical variation).– Estimates can be nonparametric or
parametric
Estimators
• TPF=
• TPF=
• Number in sample << Number in population (at least in theory)
Number of abnormals that would be selected by CIO in the population
Number of abnormals in the population
Number of abnormals that were selected by CIO in the sample
Number of abnormals in the sample
II. Receiver Operating Characteristic (ROC)
• Receiver Operating Characteristic
• Binary Classification
• Test result is compared to a threshold
Distribution of CIO Output for all Subjects
Threshold
Computational intelligence observer output
Threshold
Computational intelligence observer outputt-axis
Distribution of Outputfor Normal / Class 0
Subjects, p(t|0)
Distribution of Outputfor Abnormal / Class 1
Subjects, p(t|1)
Abnormal / Class 1subjects
Threshold
Distribution of Outputfor Normal / Class 0
Subjects, p(t|0)
Abnormal / Class 1subjects
Threshold
Sensitivity
Specificity
= True Negative Fraction = TNF
= True Positive Fraction = TPF
Distribution of Outputfor Normal / Class 0
Subjects, p(t|0)
DecisionD0 D1
Threshold
Sensitivity
Specificity
TNF0.50
TPF0.95
Tru
thH
1
H0
Normal / Class 0subjects
Abnormal / Class 1subjects
Threshold
1 - Specificity
= False Positive Fraction = FPF
1 - Sensitivity
= False Negative Fraction = FNF
Normal / Class 0subjects
Abnormal / Class 1subjects
1 - Specificity
1 - Sensitivity
FNF0.05
DecisionD0 D1
TNF0.50
FPF0.50
TPF0.95
Tru
thH
1
H0
Threshold
Normal / Class 0subjects
Abnormal / Class 1subjects
TP
F,
sens
itivi
tyFPF, 1-specificity
highsensitivity
Threshold
Normal / Class 0subjects
Abnormal / Class 1subjects
TP
F,
sens
itivi
tyFPF, 1-specificity
sensitivity = specificity
Threshold
Normal / Class 0subjects
Abnormal / Class 1subjects
TP
F,
sens
itivi
tyFPF, 1-specificity
highspecificity
Threshold
Normal / Class 0subjects
Abnormal / Class 1subjects
TP
F,
sens
itivi
tyFPF, 1-specificity
CIO #1Threshold
Normal / Class 0subjects
Abnormal / Class 1subjects
CIO #2
CIO #3
Which CIO is best?
TPF FPF
CIO #1 0.50 0.07
CIO #2 0.78 0.22
CIO #3 0.93 0.50
TP
F,
sens
itivi
tyFPF, 1-specificity
CIO #1Threshold
Normal / Class 0subjects
Abnormal / Class 1subjects
CIO #2
CIO #3
Do not compare rates of one class, e.g. TPF, at different rates of the other class (FPF).
TPF FPF
CIO #1 0.50 0.07
CIO #2 0.78 0.23
CIO #3 0.93 0.50
Threshold
TP
F,
sens
itivi
tyFPF, 1-specificity
Entire ROC curve
Normal / Class 0subjects
Abnormal / Class 1subjects
TP
F,
sens
itivi
tyFPF, 1-specificity
Entire ROC curve
Discriminability-or-
CIO performance
chan
ce lin
e
AUC=0.5
AUC=0.85
AUC=0.98
AUC (Area under ROC Curve)
• AUC is a separation probability• AUC = probability that
– CIO output for abnormal > CIO output for normal– CIO correctly tells which of 2 subjects is normal
• Estimating AUC from finite sample– Select abnormal subject score = xi
– Select normal subject score = yk
– Is xi > yk ?
– Average over all x,y:
01
I1
AUC01
n
kki
n
i
yxnn
ROC as a Q-Q plot
• ROC plots in probability space
• ROC plots in quantile space
Linear Likelihood Ratio Observer for Gaussian Data
• When the input features of the data are distributed as Gaussians with equal variance,– The optimal discriminant, the log-likelihood ratio, is a
linear function,– That linear discriminant is also distributed as a
Gaussian,– The signal to noise ratio (SNR) is easily calculated
from the input data distributions and is a monotonic function of AUC.
• Can serve as a benchmark against which to measure CIO performance
Linear Ideal Observer
• p(x|0) probability distribution of data x for the population of normals and p(x|1) probability distribution of x for the population of abnormals with components xi independent Gaussian distributed with means 0 and i respectively and identical variances i
2
D
ii
Di ixxp
1
222/2 )2/exp()2()0|(
D
iii
Di ixxp
1
222/2 )2/)(exp()2()1|(
Maximum Likelihood CIO
)0|(/)1|()0()0|(/)1()1|()|0(/)|1( xpxkppxppxpxpxpL
D
i i
iD
i i
iixkL1
2
2
12 2
expexp
D
i i
iixt1
21 exp
D
i i
iixtt1
21)ln(
Linear Ideal Observer ROC
),0(,0)0|(1
2
2
GaussGausstpN
i i
i
,)1|( Gausstp
21
21
)2/()2/(
/))0|()1|(( ))0|()1|((
tptptptpSNR
ondistributiGaussian Cumulative );( SNRAUC
21
Likelihood Ratio = Slope of ROC
• The likelihood ratio of the decision variable t is the slope of the ROC curve:
• ROC= TPF(FPF); TPF= 1-P(t|0); FPF= 1-P(t|1)
)L()0|p(
)1|p(
)0|P(
1)| (P
FPF
TPF slope t
t
t
td
td
d
d
III. Error analysis for CI observers
• Sources of error
• Parametric methods
• Nonparametric methods
• Standard deviations and confidence intervals
• Hazards
Sources of error
• Test error—limited number of samples in the test set
• Training error—limited number of samples in the training set– Incorrect parameters– Incorrect feature selection, etc.
• Human observer error (when applicable)– Intraobserver– Interobserver
Parametric methods
• Use known underlying probability distribution – may be exact for simulated data
• Assume Gaussian distribution
• Other parameterization – e.g., Binomial or ROC linearity in z-transformation coordinates– (-1(TPF) versus -1(FPF), where is the cumulative
Gaussian distribution)
Binomial Estimates of Variance
• For single population measures, f= TPF, FPF, FNF, TNF
• Var(f) = f (1-f) / N
• For AUC (back of envelope calculation)
Var(AUC) = N
AUC)-(1 AUC
Data rich case
• Repeat experiment M times
• Estimate distribution parameters—e.g., for a Gaussian distributed performance measure f, G(,2):
• Find error bars or confidence limits
Mff/ˆˆ 22
ˆ
M
iif f
M 1
22 )ˆ(1
1ˆ
M
iiMf ff
1
1ˆˆ
ff ˆˆˆ
fkf ˆ
ˆ )96.1( %95 k
Example: AUC
• Mean AUC
• “Distribution” variance
• Variance of mean
• Error bars, confidence interval
Probability distribution for calculation of AUC from 40 values
Probability distribution for calculation of SNR from 40 values
But what’s a poor boy to do?
• Reuse the data you have: Resubstitution, Resampling
• Two common approaches:– Jackknife– Bootstrap
Resampling
Resampling
Resampling
Resampling
Resampling
Resampling
Resampling
Resampling
Jackknife
• Have N observations• Leave out m of these, then have M subsets
of the N observations to calculate , 2
• N=10, m=5: M=252; N=10, m=1: M=10
m
NM
Round-robin jackknife bias derivation and variance
• Given N datasets
1
1
1
1
)1(ˆ
)1(
1
)1(
)12(2
)1/(
/
NNJ
NN
NN
N
N
AUCNAUCNCUA
AUCAUCNN
k
NN
NkAUCAUCAUC
NkAUCAUC
NkAUCAUC
21
1)(1
2 )(1
ˆ
N
N
iiNJ AUCAUC
N
N
Fukunaga-Hayes bias derivation
• Divide both the normal and abnormal classes in half, yielding 4 possible pairings
2/
2/
2ˆ
/2
/
NNHF
N
N
AUCAUCCUA
NkAUCAUC
NkAUCAUC
Jackknife bias correction exampleTraining error
• AUC estimates as a function of number of cases N. Solid line is the multilayer perceptron result. Open circle jackknife, closed circle Fukunaga-Hayes. The horizontal dotted line is the asymptotic ideal result
IV. Bootstrap methods
• Theoretical foundation
• Practical use
Bootstrap variance
• What you have is what you’ve got—the data is your best estimate of the probability distribution:– Sampling with replacement, M times– Adequate number of samples M>N
Simple bootstrap
2
1)(
2 )(1
1ˆ B
M
iiBB AUCAUC
M
Bootstrap and jackknife error estimates
• Standard deviation of AUC: Solid line simulation results, open circles jackknife estimate, closed circles bootstrap estimate. Note how much larger the jackknife error bars are than those provided by the bootstrap method.
Comparison of s.d. estimates:2 Gaussian dist., 20 normal, 20 abnormal, pop. AUC=.936
• Actual s.d. .0380• Binomial approx. .0380• Bootstrap .0388• Jackknife .0396
• Mean bootstrap AUC est. .936• Mean jackknife bias est. 2x10-17
.632 bootstrap for classifier performance evaluation
• Have N cases, draw M samples of size N with replacement for training (Have on average .632 x N unique cases in each sample of size N)
• Test on the unused (~.368 x N) cases for each sample
Ncasescases
epp
ep
testingtraining
casecase
NNcase
ii
i
...632.1
...632.11
)1(1
11
NN )/11(
N 5 10 20 100 Infinity
.3281-.672
.3491-.651
.3581-.642
.3661-.634
.3681-.632
.632 bootstrap for classifier performance evaluation 2
• Have N cases, draw M samples of size N with replacement for training (Have on average .632 x N unique cases in each sample of size N)
• Test on the unused (~.368 x N) cases for each sample
• Get bootstrap average result AUCB • Get resubstitution result (testing on training set)
AUCR
• AUC.632 = .632 x AUCB + .368 x AUCR • As variance take the AUCB variance
Traps for the unwary: Overparameterization
• Cover’s theorem:
• For N<2(d+1) a hyperplane exists that will perfectly separate almost all possible dichotomies of N points in d space
.1
2),(0
d
k k
NdNC
fd(N) for d=1,5,25,125, and the limit of large d. The abscissa x=N/2(d+1) is scaled so that the values of fd(N)=0.5 lie superposed at
x=1 for all d.
Poor data hygiene
• Reporting on training data results/ testing on training data
• Carrying out any part of the training process on data later used for testing– e.g., using all of the data to select a
manageable feature set from among a large number of features—and then dividing the data into training and test sets.
Overestimate of AUC frompoor data hygiene
0.4 0.5 0.6 0.7 0.8 0.9 10
10
20
30
40
50
60
Area Under the ROC curve
Num
ber
of e
xper
imen
ts (
out
of 9
00)
Distributions of AUC values in 900 simulation experiments (on the left) and the mean ROC curves (on the right) for four validation methods: Method 1 – Feature selection and classifier training on one dataset and classifier testing on another independent dataset; Method 2 – Given perfect feature selection, classifier training on one dataset and classifier testing on another independent dataset; Method 3 – Feature selection using the entire dataset and then the dataset is partitioned into two, one for training and one for testing the classifier; Method 4 – Feature selection, classifier training, and testing using the same dataset.
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
False Positive Fraction
Tru
e P
ositi
ve F
ract
ion
Method 1, AUC=0.52
Method 2, AUC=0.62
Method 3, AUC=0.82
Method 4, AUC=0.91
Correct feature selection is hard to do
100
101
102
103
0
20
40
60
80
100
120
140
160
180
Feature Index
Num
ber
of e
xper
imen
ts (
out
of 9
00)
0 2 4 6 8 10 120
50
100
150
200
250
Number of useful features (out of 30)
Num
ber
of e
xper
imen
ts (
out
of 9
00)
An insight of feature selection performance in Method 1. On the left plots the number of experiments (out of 900) that a feature is selected. By design of the simulation population, the first 30 features are useful for classification and the remaining are useless. On the right plots the distribution of the number of useful features (out of 30) in the 900 experiments.
Conclusions
• Accuracy and other prevalence dependent measures are inadequate
• ROC/AUC provide good measures of performance
• Uncertainty must be quantified
• Bootstrap and jackknife techniques are useful methods
V. References• [1] K. Fukunaga, Statistical Pattern Recognition, 2nd Edition. Boston: Harcourt Brace Jovanovich, 1990.• [2] K. Fukunaga and R. R. Hayes, “Effects of sample size in classifier design,” IEEE Trans. Pattern Anal. Machine Intell.,
vol. PAMI-11, pp. 873–885, 1989.• [3] D. M. Green and J. A. Swets, Signal Detection Theory and Psychophysics. New York: John Wiley & Sons, 1966.• [4] J. P. Egan, Signal Detection Theory and ROC Analysis. New York: Academic Press, 1975.• [5] C. E. Metz, “Basic principles of roc analysis,” Seminars in Nuclear Medicine, vol. VIII, no. 4.• [6] H. H. Barrett and K. J. Myers, Foundations of Image Science. Hoboken: John Wiley & Sons, 2004, ch. 13 Statistical
Decision Theory.• [7] B. Efron and R. J. Tibshirani, Introduction to the Bootstrap. Boca Raton: Chapman & Hall/CRC, 1993.• [8] B. Efron, The Jackknife, the Bootstrap and Other Resampling Plans. Philadelphia: Society for Industrial and Applied
Mathematics, 1982.• [9] A. C. Davison and D. V. Hinkley, Bootstrap Methods and their Applications. Cambridge: Cambridge University Press,
1997.• [10] B. Efron, “Estimating the error rate of a prediction rule: Some improvements on cross-validation,” Journal of the
American Statistical Association, vol. 78, pp. 316–331, 1983.• [11] B. Efron and R. J. Tibshirani, “Improvements on cross-validation: The .632+ bootstrap method,” Journal of the
American Statistical Association, vol. 92, no. 438, pp. 548–560, 1997.• [12] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 3rd Edition. New York: Springer,
2009.• [13] C. M. Bishop, Pattern Recognition and Machine Learning. New York: Springer, 2006.• [14] ——, Neural Networks for Pattern Recognition. Oxford: Oxford University Press, 1995.• [15] R. F. Wagner, D. G. Brown, J.-P. Guedon, K. J. Myers, and K. A. Wear, “Multivariate Guassian pattern classification:
effects of finite sample size and the addition of correlated or noisy features on summary measures of goodness,” in Information processing in Medical Imaging, Proceedings of IPMI ’93, 1993, pp. 507–524.
• [16] ——, “On combining a few diagnostic tests or features,” in Proceedings of the SPIE, Image Processing, vol. 2167, 1994.
• [17] D. G. Brown, A. C. Schneider, M. P. Anderson, and R. F. Wagner, “Effects of finite sample size and correlated noisy input features on neural network pattern classification,” in Proceedings of the SPIE, Image Processing, vol. 2167, 1994.
• [18] C. A. Beam, “Analysis of clustered data in receiver operating characteristic studies,” Statistical Methods in Medical Research, vol. 7, pp. 324–336, 1998.
• [19] W. A. Yousef, et al. “Assessing Classifiers from Two Independent Data Sets Using ROC Analysis: A Nonparametric Approach,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 11, pp. 1809-1817, 2006
• [19] F. W. Samuelson and D. G. Brown, “Application of cover’s theorem to the evaluation of the performance of CI observers,” in Proceedings of the IJCNN 2011, 2011.
• [20] W. Chen and D. G. Brown, “Optimistic bias in the assessment of high dimensional classifiers with a limited dataset,” in Proceedings of the IJCNN 2011, 2011.
Appendix I
Searching suitcases
Previous class results