Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR...
-
Upload
sharleen-walton -
Category
Documents
-
view
223 -
download
0
Transcript of Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR...
![Page 1: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/1.jpg)
Object Orie’d Data Analysis, Last Time
• Classical Discrimination (aka Classification)– FLD & GLR very attractive
– MD never better, sometimes worse
• HDLSS Discrimination– FLD & GLR fall apart
– MD much better
• Maximal Data Piling– HDLSS space is a strange place
![Page 2: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/2.jpg)
Kernel EmbeddingAizerman, Braverman and Rozoner
(1964) • Motivating idea:
Extend scope of linear discrimination,By adding nonlinear components to data
(embedding in a higher dim’al space)
• Better use of name:nonlinear discrimination?
![Page 3: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/3.jpg)
Kernel EmbeddingStronger effects for higher order polynomial embedding:
E.g. for cubic,
linear separation can give 4 parts (or fewer)
332 :,, xxxx
![Page 4: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/4.jpg)
Kernel EmbeddingGeneral View: for original data matrix:
add rows:
i.e. embed in ThenHigher sliceDimensional with aSpace hyperplane
dnd
n
xx
xx
1
111
nn
dnd
n
dnd
n
xxxx
xx
xx
xx
xx
212111
221
21
211
1
111
![Page 5: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/5.jpg)
Kernel EmbeddingEmbeddedFisher Linear Discrimination:
Choose Class 1, for any when:
in embedded space.• image of class boundaries in original
space is nonlinear• allows more complicated class regions• Can also do Gaussian Lik. Rat. (or
others) • Compute image by classifying points
from original space
dx 0
)2()1(1)2()1()2()1(10 ˆ2
1ˆ XXXXXXx wwt
![Page 6: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/6.jpg)
Kernel EmbeddingVisualization for Toy Examples:• Have Linear Disc. In Embedded Space• Study Effect in Original Data Space• Via Implied Nonlinear RegionsApproach:• Use Test Set in Original Space
(dense equally spaced grid)• Apply embedded discrimination Rule• Color Using the Result
![Page 7: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/7.jpg)
Kernel EmbeddingPolynomial Embedding, Toy Example 1:Parallel Clouds
![Page 8: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/8.jpg)
Kernel EmbeddingPolynomial Embedding, Toy Example 1:
Parallel Clouds• PC 1:
– always bad– finds “embedded greatest var.” only)
• FLD: – stays good
• GLR: – OK discrimination at data– but overfitting problems
![Page 9: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/9.jpg)
Kernel EmbeddingPolynomial Embedding, Toy Example 2:Split X
![Page 10: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/10.jpg)
Kernel EmbeddingPolynomial Embedding, Toy Example 2:
Split X
• FLD:
– Rapidly improves with higher degree
• GLR:
– Always good
– but never ellipse around blues…
![Page 11: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/11.jpg)
Kernel EmbeddingPolynomial Embedding, Toy Example 3:Donut
![Page 12: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/12.jpg)
Kernel EmbeddingPolynomial Embedding, Toy Example 3:
Donut
• FLD: – Poor fit for low degree
– then good
– no overfit
• GLR: – Best with No Embed,
– Square shape for overfitting?
![Page 13: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/13.jpg)
Kernel Embedding
Drawbacks to polynomial embedding:
• too many extra terms create
spurious structure
• i.e. have “overfitting”
• HDLSS problems typically get worse
![Page 14: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/14.jpg)
Kernel EmbeddingHot Topic Variation: “Kernel Machines”
Idea: replace polynomials by
other nonlinear functions
e.g. 1: sigmoid functions from neural nets
e.g. 2: radial basis functions
Gaussian kernels
Related to “kernel density estimation”
(recall: smoothed histogram)
![Page 15: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/15.jpg)
Kernel EmbeddingRadial Basis Functions:
Note: there are several ways to embed:
• Naïve Embedding (equally spaced grid)
• Explicit Embedding (evaluate at data)
• Implicit Emdedding (inner prod. based)
(everybody currently does the latter)
![Page 16: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/16.jpg)
Kernel EmbeddingNaïve Embedding, Radial basis
functions:
At some “grid points” ,
For a “bandwidth” (i.e. standard dev’n) ,
Consider ( dim’al) functions:
Replace data matrix with:
kgg ,...,
1
d
kgxgx ,...,
1
knk
n
gXgX
gXgX
1
111
![Page 17: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/17.jpg)
Kernel EmbeddingNaïve Embedding, Radial basis
functions:
For discrimination:
Work in radial basis space,
With new data vector ,
represented by:
10
10
gX
gX
0X
![Page 18: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/18.jpg)
Kernel EmbeddingNaïve Embedd’g, Toy E.g. 1: Parallel
Clouds
• Good
at data
• Poor
outside
![Page 19: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/19.jpg)
Kernel EmbeddingNaïve Embedd’g, Toy E.g. 2: Split X
• OK at
data
• Strange
outside
![Page 20: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/20.jpg)
Kernel EmbeddingNaïve Embedd’g, Toy E.g. 3: Donut
• Mostly
good
• Slight
mistake
for one
kernel
![Page 21: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/21.jpg)
Kernel Embedding
Naïve Embedding, Radial basis
functions:
Toy Example, Main lessons:
• Generally good in regions with data,
• Unpredictable where data are sparse
![Page 22: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/22.jpg)
Kernel EmbeddingToy Example 4: Checkerboard
VeryChallenging!
LinearMethod?
PolynomialEmbedding?
![Page 23: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/23.jpg)
Kernel EmbeddingToy Example 4: Checkerboard
Polynomial Embedding:
• Very poor for linear
• Slightly better for higher degrees
• Overall very poor
• Polynomials don’t have needed
flexibility
![Page 24: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/24.jpg)
Kernel EmbeddingToy Example 4: CheckerboardRadialBasisEmbedding+ FLDIsExcellent!
![Page 25: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/25.jpg)
Kernel EmbeddingDrawbacks to naïve embedding:
• Equally spaced grid too big in high d
• Not computationally tractable (gd)
Approach:
• Evaluate only at data points
• Not on full grid
• But where data live
![Page 26: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/26.jpg)
Kernel EmbeddingOther types of embedding:
• Explicit
• Implicit
Will be studied soon, after
introduction to Support Vector Machines…
![Page 27: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/27.jpg)
Kernel Embedding generalizations of this idea to other
types of analysis
& some clever computational ideas.
E.g. “Kernel based, nonlinear Principal
Components Analysis”
Ref: Schölkopf, Smola and Müller
(1998)
![Page 28: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/28.jpg)
Support Vector MachinesMotivation:
• Find a linear method that “works well”for embedded data
• Note: Embedded data are very non-Gaussian
• Suggests value ofreally new approach
![Page 29: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/29.jpg)
Support Vector MachinesClassical References:
• Vapnik (1982)
• Boser, Guyon & Vapnik (1992)
• Vapnik (1995)
Excellent Web Resource:
• http://www.kernel-machines.org/
![Page 30: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/30.jpg)
Support Vector MachinesRecommended tutorial:
• Burges (1998)
Recommended Monographs:
• Cristianini & Shawe-Taylor (2000)
• Schölkopf & Alex Smola (2002)
![Page 31: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/31.jpg)
Support Vector MachinesGraphical View, using Toy Example:
• Find separating plane
• To maximize distances from data to plane
• In particular smallest distance
• Data points closest are called
support vectors
• Gap between is called margin
![Page 32: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/32.jpg)
SVMs, Optimization Viewpoint
Formulate Optimization problem, based on:
• Data (feature) vectors • Class Labels • Normal Vector • Location (determines intercept) • Residuals (right side) • Residuals (wrong side) • Solve (convex problem) by quadratic
programming
nxx ,...,1
1iyw
b bwxyr tiii
ii r
![Page 33: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/33.jpg)
SVMs, Optimization Viewpoint
Lagrange Multipliers primal formulation (separable case):
• Minimize: Where are Lagrange
multipliers
Dual Lagrangian version:• Maximize:
Get classification function:
n
iiiiP bwxywbwL
1
2
21 1,,
0,...,1 n
i ji
jijijiiD xxyyL,
21
n
iiii bxxyxf
1
![Page 34: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/34.jpg)
SVMs, ComputationMajor Computational Point:• Classifier only depends on data
through inner products!• Thus enough to only store inner
products• Creates big savings in optimization• Especially for HDLSS data• But also creates variations in kernel
embedding (interpretation?!?)• This is almost always done in practice
![Page 35: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/35.jpg)
SVMs, Comput’n & Embedding
For an “Embedding Map”,
e.g.
Explicit Embedding:
Maximize:
Get classification function:
• Straightforward application of embedding
• But loses inner product advantage
x
2x
xx
i ji
jijijiiD xxyyL,
21
n
iiii bxxyxf
1
![Page 36: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/36.jpg)
SVMs, Comput’n & EmbeddingImplicit Embedding:
Maximize:
Get classification function:
• Still defined only via inner products• Retains optimization advantage• Thus used very commonly• Comparison to explicit embedding?• Which is “better”???
i ji
jijijiiD xxyyL,
21
n
iiii bxxyxf
1
![Page 37: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/37.jpg)
SVMs & RobustnessUsually not severely affected by outliers,But a possible weakness:
Can have very influential pointsToy E.g., only 2 points drive SVMNotes:• Huge range of chosen hyperplanes• But all are “pretty good discriminators”• Only happens when whole range is
OK???• Good or bad?
![Page 38: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/38.jpg)
SVMs & RobustnessEffect of violators (toy example):
• Depends on distance to plane
• Weak for violators nearby
• Strong as they move away
• Can have major impact on plane
• Also depends on tuning parameter C
![Page 39: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/39.jpg)
SVMs, Computation Caution: available algorithms are not
created equal
Toy Example:
• Gunn’s Matlab code
• Todd’s Matlab code
Serious errors in Gunn’s version, does not find real optimum…
![Page 40: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/40.jpg)
SVMs, Tuning Parameter Recall Regularization Parameter C:
• Controls penalty for violation
• I.e. lying on wrong side of plane
• Appears in slack variables
• Affects performance of SVM
Toy Example:
d = 50, Spherical Gaussian data
![Page 41: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/41.jpg)
SVMs, Tuning Parameter Toy Example:
d = 50, Spherical Gaussian dataX=Axis: Opt. Dir’n Other: SVM Dir’n• Small C:
– Where is the margin?– Small angle to optimal (generalizable)
• Large C:– More data piling– Larger angle (less generalizable)– Bigger gap (but maybe not better???)
• Between: Very small range
![Page 42: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/42.jpg)
SVMs, Tuning Parameter Toy Example:
d = 50, Spherical Gaussian data
Careful look at small C:
Put MD on horizontal axis E.g.
• Shows SVM and MD same for C small– Mathematics behind this?
• Separates for large C– No data piling for MD
![Page 43: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/43.jpg)
Distance Weighted Discrim’n
Improvement of SVM for HDLSS DataToy e.g.
(similar toearlier movie)
50d)1,0(N
2.21 20 nn
![Page 44: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/44.jpg)
Distance Weighted Discrim’n
Toy e.g.: Maximal Data Piling Direction- Perfect
Separation- Gross
Overfitting- Large Angle- Poor
Gen’ability
![Page 45: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/45.jpg)
Distance Weighted Discrim’n Toy e.g.: Support Vector Machine
Direction- Bigger Gap- Smaller Angle- Better
Gen’ability- Feels support
vectors toostrongly???
- Ugly subpops?- Improvement?
![Page 46: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/46.jpg)
Distance Weighted Discrim’n Toy e.g.: Distance Weighted
Discrimination- Addresses
these issues- Smaller Angle- Better
Gen’ability- Nice subpops- Replaces
min dist. by avg. dist.
![Page 47: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/47.jpg)
Distance Weighted Discrim’n Based on Optimization Problem:
More precisely: Work in appropriate penalty for violations
Optimization Method:Second Order Cone Programming
• “Still convex” gen’n of quad’c program’g
• Allows fast greedy solution• Can use available fast software
(SDP3, Michael Todd, et al)
n
i iw r1,
1min
![Page 48: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/48.jpg)
Distance Weighted Discrim’n 2=d Visualization:
Pushes PlaneAway FromData
All PointsHave SomeInfluence
n
i iw r1,
1min
![Page 49: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/49.jpg)
4949
UNC, Stat & OR
DWD Batch and Source AdjustmentDWD Batch and Source Adjustment
Recall from Class Meeting, 9/6/05: For Perou’s Stanford Breast Cancer Data Analysis in Benito, et al (2004)
Bioinformaticshttps://genome.unc.edu/pubsup/dwd/
Use DWD as useful direction vector to: Adjust for Source Effects
Different sources of mRNA Adjust for Batch Effects
Arrays fabricated at different times
![Page 50: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/50.jpg)
5050
UNC, Stat & OR
DWD Adj: Biological Class Colors & DWD Adj: Biological Class Colors & SymbolsSymbols
![Page 51: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/51.jpg)
5151
UNC, Stat & OR
DWD Adj: Source ColorsDWD Adj: Source Colors
![Page 52: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/52.jpg)
5252
UNC, Stat & OR
DWD Adj: Source Adj’d, PCA viewDWD Adj: Source Adj’d, PCA view
![Page 53: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/53.jpg)
5353
UNC, Stat & OR
DWD Adj: Source Adj’d, Class ColoredDWD Adj: Source Adj’d, Class Colored
![Page 54: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/54.jpg)
5454
UNC, Stat & OR
DWD Adj: S. & B Adj’d, Adj’d PCADWD Adj: S. & B Adj’d, Adj’d PCA
![Page 55: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/55.jpg)
5555
UNC, Stat & OR
Why not adjust using SVM?
Major Problem: Proj’d Distrib’al
Shape
Triangular Dist’ns (opposite skewed)
Does not allow sensible rigid shift
![Page 56: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/56.jpg)
5656
UNC, Stat & OR
Why not adjust using SVM?
Nicely Fixed by DWD
Projected Dist’ns near Gaussian
Sensible to shift
![Page 57: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/57.jpg)
5757
UNC, Stat & OR
Why not adjust by means?
DWD is complicated: value added?
Xuxin Liu example…
Key is sizes of biological subtypes
Differing ratio trips up mean
But DWD more robust
(although still not perfect)
![Page 58: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/58.jpg)
5858
UNC, Stat & OR
Twiddle ratios of subtypes
Link toMovie
![Page 59: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/59.jpg)
5959
UNC, Stat & OR
DWD in Face Recognition, I
Face Images as Data
(with M. Benito & D. Peña)
Registered using
landmarks
Male – Female Difference?
Discrimination Rule?
![Page 60: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/60.jpg)
6060
UNC, Stat & OR
DWD in Face Recognition, II
DWD Direction
Good separation
Images “make
sense”
Garbage at ends?
(extrapolation
effects?)
![Page 61: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/61.jpg)
6161
UNC, Stat & OR
DWD in Face Recognition, III
Interesting summary:
Jump between
means
(in DWD direction)
Clear separation of
Maleness vs.
Femaleness
![Page 62: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/62.jpg)
6262
UNC, Stat & OR
DWD in Face Recognition, IV
Fun Comparison:
Jump between means
(in SVM direction)
Also distinguishes
Maleness vs.
Femaleness
But not as well as
DWD
![Page 63: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/63.jpg)
6363
UNC, Stat & OR
DWD in Face Recognition, V
Analysis of difference: Project onto normals SVM has “small gap” (feels noise artifacts?) DWD “more informative” (feels real structure?)
![Page 64: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/64.jpg)
6464
UNC, Stat & OR
DWD in Face Recognition, VI
Current Work:
Focus on “drivers”:
(regions of interest)
Relation to Discr’n?
Which is “best”?
Lessons for human
perception?
![Page 65: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/65.jpg)
![Page 66: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/66.jpg)
• Fix links on face movies
![Page 67: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/67.jpg)
• Next Topics:
• DWD outcomes, from SAMSI below
• DWD simulations, from SAMSI below
• Windup from FDA04-22-02.doc– General Conclusion– Validation
• Also SVMoverviewSAMSI09-06-03.doc
![Page 68: Object Orie’d Data Analysis, Last Time Classical Discrimination (aka Classification) –FLD & GLR very attractive –MD never better, sometimes worse HDLSS.](https://reader030.fdocuments.net/reader030/viewer/2022032703/56649f575503460f94c7cc61/html5/thumbnails/68.jpg)
• Multi-Class SVMs• Lee, Y., Lin, Y. and Wahba, G. (2002)
"Multicategory Support Vector Machines, Theory, and Application to the Classification of Microarray Data and Satellite Radiance Data", U. Wisc. TR 1064.
• So far only have “implicit” version• “Direction based” variation is unknown