Post on 16-Oct-2020
SUPPORT VECTOR MACHINE IN MATLAB
Weicheng Guo
PROBLEM DEFINITION
2
We classify red points (in a circle with radius of 1) as 1 and green points (in a ring with radius from 1 to 2) as -1 in this figure. We want to find the hyperplane separating these two classes. Our decision boundary will have equation:
π€π€ππx+b=0
FINDING THE DECISION BOUNDARY
3
Let {x1,β¦, xn} be the training data set and let yiβ{1,-1} be the class label of xi
The decision boundary should classify all points correctly with
The decision boundary can be found by solving the following constrained optimization problem
π¦π¦ππ π€π€πππ₯π₯ππ + ππ β₯ 1 β ππππ
minββ12βπ€π€β2 + πΆπΆοΏ½ππππ
ππ
ππ=1π π π π πππ π π π π π π π ββββπ π π‘π‘ββββπ¦π¦ππ π€π€πππ₯π₯ππ + ππ β₯ 1 β ππππ
REFORMULATING AS A LAGRANGIAN
4
We can introduce Lagrange multipliers to represent the condition and thus have the following formulation:
j1 1 1
1
1max x x2
subject to 0 , 0
n n nT
i i j i j ii i j
n
i i ii
y y
C y
Ξ± Ξ±Ξ±
Ξ± Ξ±
= = =
=
β
β€ β€ =
β ββ
β
KERNEL TRICK
5
Since it is a nonlinear classification in this case, we can use kernel functions that operates on the lower dimension vectors xi and xj to produce a value equivalent to the dot product of the higher-dimensional vectors. Therefore, the data can be linearly separated in that space and the formulation is transformed to:
1 1 1
1
1max ( , )2
subject to 0 , 0
n n n
i i j i j i ji i j
n
i i ii
y y K x x
C y
Ξ± Ξ±Ξ±
Ξ± Ξ±
= = =
=
β
β€ β€ =
β ββ
β
KERNEL FUNCTIONS
6
Linear kernel
Polynomial kernel with Ξ³, d and r
Radial basis function kernel with Ξ³
Sigmoid kernel with Ξ³ and r
πΎπΎ(π₯π₯ππ , π₯π₯ππ) = π₯π₯πππππ₯π₯ππ
πΎπΎ(π₯π₯ππ , π₯π₯ππ) = πΎπΎπ₯π₯πππππ₯π₯ππ + r ππ
πΎπΎ π₯π₯ππ , π₯π₯ππ = exp(βπΎπΎ||π₯π₯ππ β π₯π₯ππ||2)
πΎπΎ π₯π₯ππ , π₯π₯ππ = tanh(πΎπΎπ₯π₯πππππ₯π₯ππ + r)
LIBSVM -- A LIBRARY FOR SUPPORT VECTOR
7
LIBSVM[1] is an open source machine learning library developed at the National Taiwan University and written in C++ though with a C API. LIBSVM implements the SMO algorithm for kernelized support vector machines (SVMs), supporting classification and regression. [1] https://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html
Differences between LIBSVM and SVM function in Matlab
LIBSVM SVM function in Matlab
Classification or regression Both Classification
Classification type Two-class and multi-class Two-class
SVM formulations C-SVC, nu-SVC, one-class SVM, epsilon-SVR, nu-SVR C-SVC
LIBSVM -- A LIBRARY FOR SUPPORT VECTOR
8
PROCEDURE 1. Transform data to the format of an SVM package
2. Conduct simple scaling on the data
3. Consider the RBF kernel K(π₯π₯ππ,π₯π₯ππ) = exp(βΞ³| π₯π₯ππ β π₯π₯ππ |2)
4. Find the best parameter C and Ξ³
5. Use the best parameter C and Ξ³ to train the whole training data
6. Test
TRAINING RESULT
9
GAMMA=0.1
TRAINING RESULT
10
GAMMA=3
TRAINING RESULT
11
GAMMA=50
TESTING RESULT
12
GAMMA=3
TESTING RESULT
13
GAMMA=3
CONCLUSION
14
SVM is a new method of machine learning based on statistics theory. In contrast to βblack boxβ learning approaches (artificial neural network), SVM is supported by certain mathematical models.
The training of SVM is relatively easy. Unlike in neural network, SVM can get global optimum and the training time does not depend on dimensionality of feature space any more by using the kernel trick.
The key to training SVM is to select a kernel function and its parameters, which cannot be conducted by a principled manner.