Introduction to Machine Learning Manik Varma Microsoft Research India manik [email protected].
-
date post
20-Dec-2015 -
Category
Documents
-
view
253 -
download
3
Transcript of Introduction to Machine Learning Manik Varma Microsoft Research India manik [email protected].
Introduction to Machine Learning
Manik VarmaMicrosoft Research India
http://research.microsoft.com/[email protected]
Binary Classification
• Is this person Madhubala or not?• Is this person male or female?• Is this person beautiful or not?
Multi-Class Classification
• Is this person Madhubala, Lalu or Rakhi Sawant?• Is this person happy, sad, angry or bemused?
Ordinal Regression
• Is this person very beautiful, beautiful, ordinary or ugly?
Regression
• How beautiful is this person on a continuous scale of 1 to 10? 9.99?
Ranking
• Rank these people in decreasing order of attractiveness.
Multi-Label Classification
• Tag this image with the set of relevant labels from {female, Madhubala, beautiful, IITD faculty}
Are These Problems Distinct?• Can regression solve all these problems• Binary classification – predict p(y=1|x)• Multi-Class classification – predict p(y=k|x)• Ordinal regression – predict p(y=k|x)• Ranking – predict and sort by relevance• Multi-Label Classification – predict p(y{1}k|x)• Learning from experience and data• In what form can the training data be obtained?• What is known a priori?• Complexity of training• Complexity of prediction
In This Course• Supervised learning
Classification• Generative methods• Nearest neighbour, Naïve Bayes
• Discriminative methods• Logistic Regression
• Discriminant methods• Support Vector Machines
Regression, Ranking, Feature Selection, etc. Unsupervised learning Semi-supervised learning Reinforcement learning
Learning from Noisy Data• Noise and uncertainty• Unknown generative model Y = f(X)• Noise in measuring input and feature extraction• Noise in labels• Nuisance variables• Missing data• Finite training set size
Under and Over Fitting
Probability Theory• Non-negativity and unit measure• 0 ≤ p(y) , p() = 1, p() = 0• Conditional probability – p(y|x)• p(x, y) = p(y|x) p(x) = p(x|y) p(y)• Bayes’ Theorem• p(y|x) = p(x|y) p(y) / p(x) • Marginalization• p(x) = y p(x, y) dy • Independence• p(x1, x2) = p(x1) p(x2) p(x1|x2) = p(x1)
Chris Bishop, “Pattern Recognition & Machine Learning”
The Univariate Gaussian Density
• p(x|,) = exp( -(x – )2/22) / (22)½
1-1 2-3 3-2 0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
x
p(x
)Univariate Gaussian
The Multivariate Gaussian Density
• p(x|,) = exp( -½ (x – )t -1 (x – ) )/ (2)D/2||½
-4-2
02
4
-2
0
20
0.1
0.2
0.3
0.4
x
Surface Plot
y
p(x
,y)
x
y
Contour Plot p(x,y)
-2 -1 0 1 2-1.5
-1
-0.5
0
0.5
1
1.5
The Beta Density
• p(|a,b) = a-1(1 – )b-1 (a+b) / (a)(b)
0 0.5 10
1
2
3
x
p(x
)
(0.1,0.1)
0 0.5 10
1
2
3
x
p(x
)
(1.0,1.0)
0 0.5 10
1
2
3
x
p(x
)
(2.0,2.0)
0 0.5 10
1
2
3
x
p(x
)
(8.0,4.0)
Probability Distribution Functions• Bernoulli: Single trial with probability of success =• n {0, 1}, [0, 1]• p(n|) = n(1 – )1-n
• Binomial: N iid Bernoulli trials with n successes• n {0, 1, …, N}, [0, 1], • p(n|N,) = NCn n(1 – )N-n
• Multinomial: N iid trials, outcome k occurs nk times• nk {0, 1, …, N}, k nk = N, k [0, 1], k k = 1• p(n|N,) = N! k k
nk / nk!
A Toy Example• We don’t know whether a coin is fair or not. We are told that heads occurred n times in N coin flips.
• We are asked to predict whether the next coin flip will result in a head or a tail.
• Let y be a binary random variable such that y = 1 represents the event that the next coin flip will be a head and y = 0 that it will be a tail
• We should predict heads if p(y=1|n,N) > p(y=0|n,N)
The Maximum Likelihood Approach• Let p(y=1|n,N) = and p(y=0|n,N) = 1 - so that we should predict heads if > ½
• How should we estimate ?
• Assuming that the observed coin flips followed a Binomial distribution, we could choose the value of that maximizes the likelihood of observing the data
• ML = argmax p(n|) = argmax NCn n(1 – )N-n
= argmax n log() + (N – n) log(1 – ) = n / N
• We should predict heads if n > ½ N
The Maximum A Posteriori Approach• We should choose the value of maximizing the posterior probability of conditioned on the data
• We assume a• Binomial likelihood : p(n|) = NCn n(1 – )N-n • Beta prior : p(|a,b)=a-1(1–)b-1(a+b)/(a)(b)
• MAP = argmax p(|n,a,b) = argmax p(n|) p(|a,b) = argmax n (1 – )N-n a-1 (1–)b-1
= (n+a-1) / (N+a+b-2) as if we saw an extra a – 1 heads & b – 1 tails
• We should predict heads if n > ½ (N + b – a)
The Bayesian Approach• We should marginalize over
• p(y=1|n,a,b) = p(y=1|n,) p(|a,b,n) d= p(|a,b,n) d= (|a + n, b + N – n) d= (n + a) / (N + a + b) as if we saw an extra a heads & b tails
• We should predict heads if n > ½ (N + b – a)
• The Bayesian and MAP prediction coincide in this case
• In the very large data limit, both the Bayesian and MAP prediction coincide with the ML prediction (n > ½ N)
Classification
Binary Classification
Approaches to Classification• Memorization• Can not deal with previously unseen data• Large scale annotated data acquisition cost might
be very high• Rule based expert system• Dependent on the competence of the expert.• Complex problems lead to a proliferation of rules,
exceptions, exceptions to exceptions, etc.• Rules might not transfer to similar problems• Learning from training data and prior knowledge• Focuses on generalization to novel data
Notation• Training Data• Set of N labeled examples of the form (xi, yi)• Feature vector – x D. X = [x1 x2 … xN]• Label – y {1}. y = [y1, y2 … yN]t. Y=diag(y)• Example – Gender Identification
(x1 = , y1 = +1) (x2 = , y2 = +1)
(x3 = , y3 = +1) (x4 = , y4 = -1)
Binary Classification
wtx + b = 0
b
w
Binary Classification
= [w ; b]
Bayes’ Decision Rule• Bayes’ decision rule
p(y=+1|x) > p(y=-1|x) ? y = +1 : y = -1 p(y=+1|x) > ½ ? y = +1 : y = -1
-6 -4 -2 0 2 4 60
0.5
1
1.5
x
p(y
|x)
p(y=+1|x)p(y=-1|x)
Issues to Think About• Bayesian versus MAP versus ML• Should we choose just one function to explain
the data?• If yes, should this be the function that explains
the data the best?• What about prior knowledge?• Generative versus Discriminative• Can we learn from “positive” data alone?• Should we model the data distribution?• Are there any missing variables?• Do we just care about the final decision?
Bayesian Approachp(y|x,X,Y) = f p(y,f|x,X,Y) df
= f p(y|f,x,X,Y) p(f|x,X,Y) df= f p(y|f,x) p(f|X,Y) df
• This integral is often intractable. • To solve it we can• Choose the distributions so that the solution is
analytic (conjugate priors)• Approximate the true distribution of p(f|X,Y) by a
simpler distribution (variational methods)• Sample from p(f|X,Y) (MCMC)
Maximum A Posteriori (MAP)p(y|x,X,Y) = f p(y|f,x) p(f|X,Y) df
= p(y|fMAP,x) when p(f|X,Y) = (f – fMAP)
• The more training data there is the better p(f|X,Y) approximates a delta function• We can make predictions using a single function, fMAP, and our focus shifts to estimating fMAP.
f
p(f
|X,Y
)
No DataModerate DataLots of Data
MAP & Maximum Likelihood (ML)fMAP = argmaxf p(f|X,Y)
= argmaxf p(X,Y|f) p(f) / p(X,Y) = argmaxf p(X,Y|f) p(f)
fML argmaxf p(X,Y|f) (Maximum Likelihood)
• Maximum Likelihood holds if• There is a lot of training data so that
p(X,Y|f) >> p(f)• Or if there is no prior knowledge so that p(f) is
uniform (improper)
IID Data fML = argmaxf p(X,Y|f)
= argmaxf I p(xi,yi|f)
• The independent and identically distributed assumption holds only if we know everything about the joint distribution of the features and labels. In particular, p(X,Y) I p(xi,yi)
Generative MethodsNaïve Bayes
Generative Methods MAP = argmax p() I p(xi,yi| )
= argmax p(x) p(y) I p(xi,yi| )= argmax p(x) p(y) I p(xi|yi,) p(yi|)= argmax p(x) p(y) I p(xi|yi,) p(yi|)= [argmaxx p(x) I p(xi|yi,x)] * [argmaxy p(y) I p(yi|x)]
• x and y can be solved for independently• The parameters of each class decouple and can be solved for independently
Generative Methods• The parameters of each class decouple and can be solved for independently
Generative Methods – Naïve Bayes MAP = [argmaxx p(x) I p(xi|yi,x)] *
[argmaxy p(y) I p(yi|x)]
• Naïve Bayes assumptions• Independent Gaussian features
p(xi|yi,x) = j p(xij|yi,x)p(xij|yi=1,x) = N(xij| j
1, i)• Improper uniform priors (no prior knowledge)
p(x) = p(y) = const• Bernoulli labels
p(yi=+1|y) = , p(yi=-1|y) = 1-
Generative Methods – Naïve Bayes ML = [argmaxx I j N(xij| j
1, i)] * [argmax I (1+yi)/2 (1-)(1-yi)/2]
• Estimating ML
ML = argmax I (1+yi)/2 (1-)(1-yi)/2 = argmax (N+I yi) log() + (N-I yi) log(1-)
= N+ / N (by differentiating and setting to zero)
• Estimating ML, ML
ML = (1 / N) yi=1
xi
2jML = [ yi=+1 (xij - +
jML)2 + yi=-1 (xij - -jML)2 ]/N
Naïve Bayes – Prediction
-5 0 5-5
0
5Data
-5 0 5-5
0
5Train Err = 16.67%
Decision Boundaries
20 40 60 80 100
20
40
60
80
100-5
05
-50
50
0.5
1
Posterior p(y|x)
Naïve Bayes – Prediction
-5 0 5-5
0
5Data
-5 0 5-5
0
5Train Err = 0.00%
Decision Boundaries
20 40 60 80 100
20
40
60
80
100-5
05
-50
50
0.5
1
Posterior p(y|x)
Naïve Bayes – Predictionp(y=+1|x) = p(x|y=+1) p(y=+1) / p(x)
= 1 / (1 + exp(log(p(y=-1)/ p(y=+1))+log(p(x|y=-1) / p(x|y=+1)))
= 1 / (1 + exp( log(1/ - 1) - ½ -t-1- + ½ +t-1+ + (+ - -) t-1x ))
= 1 / (1 + exp(-b – wtx)) (Logistic Regression)
Þ p(y=-1|x) = exp(-b – wtx) / (1 + exp(-b – wtx))Þ log(p(y=-1|x)/ p(y=+1|x)) = -b – wtxÞ y = sign(b + wtx)Þ The decision boundary will be linear!
Discriminative Methods Logistic Regression
Discriminative Methods MAP = argmax p() I p(xi,yi| )• We assume that• p() = p(w) p(w)• p(xi,yi| ) = p(yi| xi, ) p(xi| )
= p(yi| xi, w) p(xi| w) Þ MAP = [argmaxw p(w) I p(yi| xi, w)] * [argmaxw
p(w) I p(xi|w)]
• It turns out that only w plays no role in determining the posterior distributionÞ p(y|x,X,Y) = p(y|x, MAP) = p(y|x, wMAP) where wMAP = argmaxw p(w) I p(yi| xi, w)
Disc. Methods – Logistic RegressionMAP = argmaxw,b p(w) I p(yi| xi, w)
• Regularized Logistic Regression• Gaussian prior – p(w) = exp( -½ wtw)• Logistic likelihood–
p(yi| xi, w) = 1 / (1 + exp(-yi(b + wtxi)))
-6 -4 -2 0 2 4 60
0.5
1
1.5
x
p(y
|x)
p(y=+1|x)p(y=-1|x)
Regularized Logistic RegressionMAP = argmaxw,b p(w) I p(yi| xi, w)
= argminw,b ½wtw + I log(1+exp(-yi(b+wtxi)))
• Bad news: No closed form solution for w and b• Good news: We have to minimize a convex function• We can obtain the global optimum• The function is smooth
Tom Minka, “A comparison of numerical optimizers for LR” (Matlab code)Keerthi et al., “A Fast Dual Algorithm for Kernel Logistic Regression”, ML 05Andrew and Gao, “OWL-QN” ICML 07Krishnapuram et al., “SMLR” PAMI 05
Regularized Logistic Regression
Zhu & Hastie, “KLR and the Import Vector Machine”, NIPS 01
-5 0 5-5
0
5Regularized Logistic Regression
Regularized Logistic Regression
Zhu & Hastie, “KLR and the Import Vector Machine”, NIPS 01
-5 0 5-5
0
5Regularized Logistic Regression
Naïve Bayes versus Logistic Regression
-5 0 5-5
0
5NB acc=100.00%, LR acc=100.00%
NBLR
Naïve Bayes versus Logistic Regression
-5 0 5-5
0
5NB acc=92.65%, LR acc=100.00%
NBLR
Naïve Bayes versus Logistic Regression
-5 0 5-5
0
5NB acc=84.85%, LR acc=87.88%
NBLR
Convex Functions
• Convex f : f(x1 + (1- )x2) f(x1) + (1- )f(x2)• The Hessian 2f is always positive semi-definite• The tangent is always a lower bound to f
0 0.5 1 1.5 2-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
x
x lo
g(x
)
Gradient Descent
• Iteration : xn+1 = xn - nf(xn)• Step size selection : Armijo rule• Stopping criterion : Change in f is “miniscule”
0.5 1 1.5 2
-1
-0.5
0
0.5
1
1.5
2
x1
x2
x3 x
4
x
x lo
g(x
)
Gradient Descent – Logistic Regression(w, b) = ½wtw + I log(1+exp(-yi(b+wtxi)))
Þw(w, b) = w – I p(-yi|xi,w) yi xi
Þ b(w, b) = – I p(-yi|xi,w) yi
Beware of numerical issues while coding!
Newton Methods
• Iteration : xn+1 = xn - nH-1f(xn)• Approximate f by a 2nd order Taylor expansion• The error can now decrease quadratically
0 0.5 1 1.5 2
-1
-0.5
0
0.5
1
1.5
2
x1
x2
x3
x
x lo
g(x
)
Quasi-Newton Methods• Computing and inverting the Hessian is expensive• Quasi-Newton methods can approximate H-1 directly (LBFGS)• Iteration : xn+1 = xn - nBn
-1f(xn)• Secant equation : f(xn+1) – f(xn) = Bn+1(xn+1 – xn)• The secant equation does not fully determine B • LBFGS updates Bn+1
-1 using two rank one matrices
Generative versus Discriminative• A discriminative model might be correct even when the corresponding generative model is not• A discriminative model has fewer parameters than the corresponding generative model• A generative models parameters are uncoupled and can often be estimated in closed form• A discriminative models parameters are correlated and training algorithms can be relatively expensive• A discriminative model often has lower test error given a “reasonable” amount of training data.• A generative model can deal with missing data
Generative versus Discriminative• Let (hA,N) denote the error of hypothesis h trained using algorithm A on N data points• When the generative model is correct
(hDis,) = (hGen,)• When the generative model is incorrect
(hDis,) (hGen,)• For a linear classifier trained in D dimensions
(hDis,N) (hDis,) + O( [-z log z]½) where z=D/N 1• It suffices to pick N = (D) points for discriminative learning of linear classifiers• For some generative models N = (log D)
Generative versus Discriminative• A generative classifier might converge much faster to its higher asymptotic error
Ng & Jordan, “On Discriminative vs. Generative Classifiers” NIPS 02.Tom Mitchell, “Generative and Discriminative Classifiers“
Multi-class Logistic Regression• Multinomial Logistic Regression• 1-vs-All• Learn L binary classifiers for an L class problem• For the lth classifier, examples from class l are +ve
while examples from all other classes are –ve• Classify new points according to max probability• 1-vs-1• Learn L(L-1)/2 binary classifiers for an L class
problem by considering every class pair• Classify novel points by majority vote• Classify novel points by building a DAG
Multi-class Logistic Regression• Assume• Non-linear multi-class classifier• Number of classes = L• Number of training points per class = N• Algorithm training time for M points = O(M3)• Classification time given M training points=O(M)
Multi-class Logistic Regression• Multinomial Logistic Regression• Training time = O(L6N3)• Classification time for a new point = O(L2N)• 1-vs-All• Training time = O(L4N3)• Classification time for a new point = O(L2N)• 1-vs-1• Training time = O(L2N3)• Majority vote classification time = O(L2N)• DAG classification time = O(LN)
Multinomial Logistic RegressionMAP = argmaxw,b p(w) I p(yi| xi, w)
• Regularized Multinomial Logistic Regression• Gaussian prior
p(w) = exp( -½ lwltwl)
• Multinomial logistic posteriorp(yi = l | xi, w) = efl(xi) / k efk(xi) where fk(xi) = wk
txi + bk
Note that we have to learn an extra classifier by not explicitly enforcing l p(yi = l | xi, w) = 1
Multinomial Logistic Regression(w, b) = ½ kwk
twk + I [log(k fk(xi)) - kkyi fk(xi)]
Þwk(w, b) = wk + I [ p(yi = k | xi,w) - kyi
] xi
Þ bk(w, b) = I [ p(yi = k | xi,w) - kyi
]
Multi-class Logistic Regression
-5 0 5-5
0
5Press class number or Esc to quit
-5 0 5-5
0
5MLR
-5 0 5-5
0
51-vs-1
-5 0 5-5
0
51-vs-All
Multi-class Logistic RegressionMLR
20 40 60 80 100
20
40
60
80
100
1-vs-All
20 40 60 80 100
20
40
60
80
100
Maj 1-vs-1
20 40 60 80 100
20
40
60
80
100
Dag 1-vs-1
20 40 60 80 100
20
40
60
80
100
Multi-class Logistic Regression
-5 0 5-5
0
5Press class number or Esc to quit
-5 0 5-5
0
5MLR
-5 0 5-5
0
51-vs-1
-5 0 5-5
0
51-vs-All
Multi-class Logistic RegressionMLR
20 40 60 80 100
20
40
60
80
100
1-vs-All
20 40 60 80 100
20
40
60
80
100
Maj 1-vs-1
20 40 60 80 100
20
40
60
80
100
Dag 1-vs-1
20 40 60 80 100
20
40
60
80
100
Multi-class Logistic Regression
-5 0 5-5
0
5Press class number or Esc to quit
-5 0 5-5
0
5MLR
-5 0 5-5
0
51-vs-1
-5 0 5-5
0
51-vs-All
Multi-class Logistic RegressionMLR
20 40 60 80 100
20
40
60
80
100
1-vs-All
20 40 60 80 100
20
40
60
80
100
Maj 1-vs-1
20 40 60 80 100
20
40
60
80
100
Dag 1-vs-1
20 40 60 80 100
20
40
60
80
100
From Probabilities to Loss FunctionsMAP = argminw,b ½wtw + I log(1+exp(1-yi(b+wtxi)))
-5 0 5
0
1
2
3
4
5
6
yf(x)
Lo
ssLoss Functions
0/1HingeSquare HingeLogRobust
Support Vector Machines
Binary Classification
A Separating Hyperplane
Maximum Margin Hyperplane
Geometric Intuition: Choose the perpendicular bisector of the shortest line segment joining the convex hulls of the two classes
wtx + b = 0
b
w
SVM Notation
wtx + b = +1
wtx + b = -1
Support Vector
Support Vector
Support Vector
Support Vector
Margin = 2 / wtw
Calculating the Margin • Let x+ be any point on the +ve supporting plane and x- the closest point on the –ve supporting plane
Margin = |x+ – x-|= |w| (since x+ = x- + w)= 2 |w|/|w|2 (assuming = 2/|w|2)= 2/|w|
wtx+ + b = +1wtx- + b = -1 wt(x+ – x-)= 2 wtw= 2 = 2/|w|2
Hard Margin SVM Primal• Maximize 2/|w| such that wtxi + b +1 if yi = +1
wtxi + b -1 if yi = -1
• Difficult to optimize directly
• Convex Quadratic Program (QP) reformulation• Minimize ½wtw such that yi(wtxi + b) 1
• Convex QPs can be easy to optimize
Linearly Inseparable Data• Minimize ½wtw + C #(Misclassified points) such that yi(wtxi + b) 1 (for “good” points)
• The optimization problem is NP Hard in general• Disastrous errors are penalized the same as near misses
wtx + b = 0
b
w
Inseparable Data – Hinge Loss
wtx + b = +1
wtx + b = -1
Support Vector
Misclassified point
Support Vector
Margin = 2 / wtw
= 0
< 1
> 1
= 0
The C-SVM Primal Formulation• Minimize ½wtw + C i i
such that yi(wtxi + b) 1 – i i 0
• The optimization is a convex QP• The globally optimal solution will be obtained• Number of variables = D + N + 1• Number of constraints = 2N • Solvers can train on 800K points in 47K (sparse) dimensions in less than 2 minutes on a standard PC
Fan et al., “LIBLINEAR” JMLR 08Bordes et al., “LaRank” ICML 07
The C-SVM Dual Formulation• Maximize 1t – ½tYKY
such that 1tY = 0 0 C
• K is a kernel matrix such that Kij = K(xi, xj) = xitxj
• are the dual variables (Lagrange multipliers)• Knowing gives us w and b • The dual is also a convex QP• Number of variables = N• Number of constraints = 2N + 1
Fan et al., “LIBSVM” JMLR 05 Joachims, “SVMLight”
SVMs versus Regularized LR
Most of the SVM s are zero!
-5 0 5-5
0
5SVM vs RLR
-5 0 5-5
0
5RLR
-5 0 5-5
0
5SVM
RLRSVM
SVMs versus Regularized LR
Most of the SVM s are zero!
-5 0 5-5
0
5SVM vs RLR
-5 0 5-5
0
5RLR
-5 0 5-5
0
5SVM
RLRSVM
SVMs versus Regularized LR
Most of the SVM s are not zero
-5 0 5-5
0
5SVM vs RLR
-5 0 5-5
0
5RLR
-5 0 5-5
0
5SVM
RLRSVM
Duality• Primal P = Minx f0(x)
s. t. fi(x) 0 1 i N hi(x) = 0 1 i M
• Lagrangian L(x,,) = f0(x) + i ifi(x) + i ihi(x)
• Dual D = Max, Minx L(x,,) s. t. 0
Duality• The Lagrange dual is always concave (even if the primal is not convex) and might be an easier problem to optimize
• Weak duality : P D • Always holds
• Strong duality : P = D • Does not always hold• Usually holds for convex problems • Holds for the SVM QP
Karush-Kuhn-Tucker (KKT) Conditions• If strong duality holds, then for x*, * and * to be optimal the following KKT conditions must necessarily hold
• Primal feasibility : fi(x*) 0 & hi(x*) = 0 for 1 i • Dual feasibility : * 0• Stationarity : x L(x*, *,*) = 0• Complimentary slackness : i*fi(x*) = 0
• If x+, + and + satisfy the KKT conditions for a convex problem then they are optimal
SVM – Duality• Primal P = Minw,,b ½wtw + Ct
s. t. Y(Xtw + b1) 1 – 0
• Lagrangian L(,, w,,b) = ½wtw + Ct – t –t[Y(Xtw + b1) – 1 + ]
• Dual D = Max 1t – ½tYKY
s. t. 1tY = 0 0 C
SVM – KKT Conditions• Lagrangian L(,, w,,b) = ½wtw + Ct – t
–t[Y(Xtw + b1) – 1 + ]
• Stationarity conditions• w L= 0 w* = XY* (Representer Theorem)• L= 0 C = * + *• b L= 0 *tY1 = 0
• Complimentary Slackness conditions• i* [ yi (xi
tw* + b*) – 1 + i*] = 0• i*i* = 0
Hinge Loss and Sparseness in • From the Stationarity and Complimentary Slackness conditions it is easy to show that
• i = 0 xi has been classified correctly and lies beyond its supporting hyperplane
• 0 < i < C xi is a support vector and lies on its supporting hyperplane
• i = C xi has been misclassified or is a margin violator
Hinge Loss and Sparseness in
• SVM s are sparse but LR s are not
-2 -1 0 1 2 3 4-0.5
0
0.5
1
1.5
2
2.5
3
3.5
4
yf(x)
Lo
ss
Loss Functions
LogHinge
Linearly Inseparable Data
• This 1D dataset can not be separated using a single hyperplane (threshold)• We need a non-linear decision boundary
x
Increasing Dimensionality Non-linearly
• The dataset is now linearly separable in space
(x) = (x, x2)x
The Kernel Trick• Let the “lifted” training set be { ((xi), yi) } • Define the kernel such that Kij = K(xi, xj) = (xi)t (xj)
• Primal P = Minw,,b ½wtw + Ct
s. t. Y((X)tw + b1) 1 – 0
• Dual D = Max 1t – ½tYKY
s. t. 1tY = 0 0 C
• Classifier: f(x) = sign((x)tw + b) = sign(tYK(:,x) + b)
The Kernel Trick• Let (x) = [1, 2x1, … , 2xD , x1
2, … , xD2, 2x1x2, …, 2x1xD, …, 2xD-1xD]t
• Define K(xi, xj) = (xi)t (xj) = (xitxj + 1)2
• Primal• Number of variables = D + N + 1• Number of constraints = 2N• Number of flops for calculating (x)tw = O(D2)• Number of flops for deg 20 polynomial = O(D20)• Dual• Number of variables = N• Number of constraints = 2N + 1• Number of flops for calculating Kij = O(D)• Number of flops for deg 20 polynomial = O(D)
Some Popular Kernels• Linear : K(xi,xj) = xi
t-1xj
• Polynomial : K(xi,xj) = (xit-1xj + c)d
• Gaussian (RBF) : K(xi,xj) = exp( –k k(xik – xjk)2)
• Chi-Squared : K(xi,xj) = exp( –2(xi, xj) )
• Sigmoid : K(xi,xj) = tanh(xitxj – c)
should be positive definite, c 0, 0 and d should be a natural number
Valid Kernels – Mercer’s Theorem• Let Z be a compact subset of D and K a continuous symmetric function. Then K is a kernel if
Z Z f(x) K(x,z) f(z) dx dz 0
for all square integrable real valued function f on Z.
Valid Kernels – Mercer’s Theorem• Let Z be a compact subset of D and K a continuous symmetric function. Then K is a kernel if
Z Z f(x) K(x,z) f(z) dx dz 0
for all square integrable real valued function f on Z.
• K is a kernel if every finite symmetric matrix formed by evaluating K on pairs of points from Z is positive semi-definite
Operations on Kernels• The following operations result in valid kernels
• K(xi,xj) = k k Kk(xi,xj) (k 0)• K(xi,xj) = k Kk(xi,xj)• K(xi,xj) = f(xi) f(xj) (f : D )• K(xi,xj) = p(K1(xi,xj)) (p : +ve coeff poly)• K(xi,xj) = exp(K1(xi,xj))
• Kernels can be defined over graphs, sets, strings and many other interesting data structures
Kernels• Kernels should encode all our prior knowledge about feature similarities.
• Kernel parameters can be chosen through cross validation or learnt (see Multiple Kernel Learning).
• Non-linear kernels can sometimes boost classification performance tremendously.
• Non-linear kernels are generally expensive (both during training and for prediction)
Polynomial Kernel of Degree 2
1 2 3 4 5
1
2
3
4
5Poly Deg 2
1 2 3 4 5
1
2
3
4
5Decision Boundaries
Abs(Distances)
2
4
6
8
Classification
Polynomial Kernel of Degree 5
1 2 3 4 5
1
2
3
4
5Poly Deg 5
1 2 3 4 5
1
2
3
4
5Decision Boundaries
Abs(Distances)
5
10
15
Classification
RBF Kernel
1 2 3 4 5
1
2
3
4
5RBF =1.000
1 2 3 4 5
1
2
3
4
5Decision Boundaries
Abs(Distances)
0.2
0.4
0.6
0.8
1
Classification
Exponential 2 Kernel
1 2 3 4 5
1
2
3
4
5EChi2 =1.000
1 2 3 4 5
1
2
3
4
5Decision Boundaries
Abs(Distances)
0.5
1
1.5
Classification
Kernel Parameter Setting - Underfitting
1 2 3 4 5
1
2
3
4
5RBF =0.001
1 2 3 4 5
1
2
3
4
5Decision Boundaries
Abs(Distances)
0.5
1
1.5
2
2.5
Classification
Kernel Parameter Setting
1 2 3 4 5
1
2
3
4
5RBF =1.000
1 2 3 4 5
1
2
3
4
5Decision Boundaries
Abs(Distances)
0.2
0.4
0.6
0.8
1
Classification
Kernel Parameter Setting – Overfitting
1 2 3 4 5
1
2
3
4
5RBF =100.000
1 2 3 4 5
1
2
3
4
5Decision Boundaries
Abs(Distances)
0.2
0.4
0.6
0.8
1
Classification
Structured Output Prediction • Minimize f ½|f|2 + C i i
such that f(xi,yi) f(xi,y) + (yi,y) – i y yi
i 0
• Prediction argmaxy f(x,y)
• This formulation minimizes the hinge on the loss on the training set subject to regularization on f Can be used to predict sets, graphs, etc. for suitable choices of Taskar et al., “Max-Margin Markov Networks” NIPS 03Tsochantaridis et al., “Large Margin Methods for Structured & Interdependent Output Variables” JMLR 05
Multi-Class SVM• Minimize f ½|f|2 + C i i
such that f(xi,yi) f(xi,y) + (yi,y) – i y yi
i 0
• Prediction argmaxy f(x,y)
• (yi,y) = yi,y
• f(x,y) = wt [ (x) (y) ] + bt(y) = wy
t(x) + by (assuming (y) = ey)
Weston and Watkin, “SVMs for Multi-Class Pattern Recognition” ESANN 99Bordes et al., “LaRank” ICML 07
Multi-Class SVM• Minw,b ½ kwk
twk + C i i
s. t. wyit(xi) + byi
wyt(xi) + by + 1 – I yyi
i 0
• Prediction argmaxy wyt(x) + by
• For L classes, with N points per class, the number of constraints is NL2
• Finding the exact solution for real world non-linear problems is often infeasible• In practice, we can obtain an approximate solution or switch to the 1-vs-All or 1-vs-1 formulations
Acknowledgements • Ankit Sagwal• Saurabh Gupta