Support Vector Machines in Data Mining AFOSR Software & Systems Annual Meeting Syracuse, NY June...
-
Upload
julianna-rose -
Category
Documents
-
view
215 -
download
1
Transcript of Support Vector Machines in Data Mining AFOSR Software & Systems Annual Meeting Syracuse, NY June...
Support Vector Machines in Data Mining
AFOSR Software & Systems Annual Meeting Syracuse, NY June 3-7, 2002
Olvi L. Mangasarian
Data Mining Institute
University of Wisconsin - Madison
What is a Support Vector Machine?
An optimally defined surface Linear or nonlinear in the input space Linear in a higher dimensional feature space Implicitly defined by a kernel function
What are Support Vector Machines Used For?
Classification Regression & Data Fitting Supervised & Unsupervised Learning
Principal ContributionsLagrangian support vector machine classification
Fast, simple, unconstrained iterative methodReduced support vector machine classification
Accurate nonlinear classifier using random samplingProximal support vector machine classification
Classify by proximity to planes instead of halfspacesMassive incremental classification
Classify by retiring old data & adding new dataKnowledge-based classification
Incorporate expert knowledge into classifierFast Newton method classifier
Finitely terminating fast algorithm for classificationBreast cancer prognosis & chemotherapy
Classify patients on basis of distinct survival curves
Support Vector MachinesMaximize the Margin between Bounding Planes
x0w = í + 1
x0w = í à 1
A+
A-
jjwjj22
w
Proximal Support Vector Machines Maximize the Margin between Proximal Planes
x0w = í + 1
x0w = í à 1
A+
A-
jjwjj22
w
Standard Support Vector MachineAlgebra of 2-Category Linearly Separable Case
Given m points in n dimensional space Represented by an m-by-n matrix A Membership of each in class +1 or –1 specified by:A i
An m-by-m diagonal matrix D with +1 & -1 entries
D(Awà eí )=e;
More succinctly:
where e is a vector of ones.
x0w = í æ1: Separate by two bounding planes,
A iw=í + 1; for D i i = + 1;A iw5í à 1; for D i i = à 1:
Standard Support Vector Machine Formulation
Margin is maximized by minimizing21kw;í k2
2
÷> 0 Solve the quadratic program for some :
2÷kyk2
2 + 21kw;í k2
2
D(Awà eí ) + y > ey;w;ímin
s. t.(QP)
,
, denoteswhere D ii = æ1 A+ Aàor membership.
PSVM Formulation
Standard SVM formulation:
w;í (QP)2÷kyk2
2 + 21kw;í k2
2
D(Awà eí ) + y
min
s. t. = e=
This simple, but critical modification, changes the nature of the optimization problem tremendously!!
Solving for in terms of and gives:
minw;í 2
÷keà D(Awà eí )k22 + 2
1kw; í k22
y w í
Advantages of New Formulation
Objective function remains strongly convex.
An explicit exact solution can be written in terms of the problem data.
PSVM classifier is obtained by solving a single system of linear equations in the usually small dimensional input space.
Exact leave-one-out-correctness can be obtained in terms of problem data.
Linear PSVM
We want to solve:
w;ímin
2÷keà D(Awà eí )k2
2 + 21kw; í k2
2
Setting the gradient equal to zero, gives a nonsingular system of linear equations.
Solution of the system gives the desired PSVM classifier.
Linear PSVM Solution
H = [A à e]Here,
íw
h i= (÷
I + H 0H)à 1H 0De
The linear system to solve depends on:
H 0H(n + 1) â (n + 1)which is of size
is usually much smaller than n m
Linear & Nonlinear PSVM MATLAB Code
function [w, gamma] = psvm(A,d,nu)% PSVM: linear and nonlinear classification% INPUT: A, d=diag(D), nu. OUTPUT: w, gamma% [w, gamma] = psvm(A,d,nu); [m,n]=size(A);e=ones(m,1);H=[A -e]; v=(d’*H)’ %v=H’*D*e; r=(speye(n+1)/nu+H’*H)\v % solve (I/nu+H’*H)r=v w=r(1:n);gamma=r(n+1); % getting w,gamma from r
Numerical experimentsOne-Billion Two-Class Dataset
Synthetic dataset consisting of 1 billion points in 10- dimensional input space Generated by NDC (Normally Distributed Clustered) dataset generatorDataset divided into 500 blocks of 2 million points each.Solution obtained in less than 2 hours and 26 minutes About 30% of the time was spent reading data from disk.Testing set Correctness 90.79%
Incoporating Knowledge Sets Into an SVM Classifier
This implication is equivalent to a set of constraints that can be imposed on the classification problem.
Suppose that the knowledge set: belongs to the class A+. Hence it must lie in the halfspace :
èx??Bx 6 b
é
èxjx0w>í + 1
é
Bx6b ) x0w>í + 1
We therefore have the implication:
Numerical TestingThe Promoter Recognition Dataset
Promoter: Short DNA sequence that precedes a gene sequence.
A promoter consists of 57 consecutive DNA nucleotides belonging to {A,G,C,T} .
Important to distinguish between promoters and nonpromoters
This distinction identifies starting locations of genes in long uncharacterized DNA sequences.
Wisconsin Breast Cancer Prognosis Dataset Description of the data
110 instances corresponding to 41 patients whose cancer had recurred and 69 patients whose cancer had not recurred
32 numerical features The domain theory: two simple rules used by doctors:
Wisconsin Breast Cancer Prognosis Dataset Numerical Testing Results
Doctor’s rules applicable to only 32 out of 110 patients.
Only 22 of 32 patients are classified correctly by this rule (20% Correctness).
KSVM linear classifier applicable to all patients with correctness of 66.4%.
Correctness comparable to best available results using conventional SVMs.
KSVM can get classifiers based on knowledge without using any data.
Newton Algorithm
f (z) = 21íí (eà D(Awà ew))+
ww2
+ 21íí w; í
íí 2
zi+1 = zi à @2f (zi)à 1r f (zi)Newton algorithm terminates in a finite number of steps
Termination at global minimum
Error rate decreases linearlyCan generate complex nonlinear classifiers
By using nonlinear kernels: K(x,y)
ConclusionNew methods for classification proposedAll based on rigorous mathematical foundationFast computational algorithms capable of classifying massive datasetsClassifiers based on both abstract prior knowledge as well as conventional datasetsIdentification of breast cancer patients that can benefit from chemotherapy
Future WorkExtend proposed methods to standard optimization problems
Linear & quadratic programming Preleminary results beat state-of-the-art software
Incorporate abstract concepts into optimization problems as constraintsDevelop fast online algorithms for intrusion and fraud detection Classify the effectiveness of new drug cocktails in combating various forms of cancer
Encouraging preliminary results