Science of Medicine Orientation Michael Meyers, MD Alisa Wolberg, PhD.
Olvi L. Mangasarian with G. M. Fung, Y.-J. Lee, J.W. Shavlik, W. H. Wolberg
-
Upload
kitra-jordan -
Category
Documents
-
view
23 -
download
1
description
Transcript of Olvi L. Mangasarian with G. M. Fung, Y.-J. Lee, J.W. Shavlik, W. H. Wolberg
Support Vector Machine ClassificationComputation & Informatics in Biology & Medicine
Madison Retreat, November 15, 2002
Olvi L. Mangasarian
with
G. M. Fung, Y.-J. Lee, J.W. Shavlik, W. H. Wolberg
& Collaborators at ExonHit – Paris
Data Mining Institute
University of Wisconsin - Madison
What is a Support Vector Machine?
An optimally defined surface Linear or nonlinear in the input space Linear in a higher dimensional feature space Implicitly defined by a kernel function K(A,B) C
What are Support Vector Machines Used For?
Classification Regression & Data Fitting Supervised & Unsupervised Learning
Principal TopicsProximal support vector machine classification
Classify by proximity to planes instead of halfspacesMassive incremental classification
Classify by retiring old data & adding new dataKnowledge-based classification
Incorporate expert knowledge into a classifierFast Newton method classifier
Finitely terminating fast algorithm for classificationBreast cancer prognosis & chemotherapy
Classify patients on basis of distinct survival curves Isolate a class of patients that may benefit from chemotherapy
Principal Topics
Proximal support vector machine classification
Support Vector MachinesMaximize the Margin between Bounding Planes
x0w = í + 1
x0w = í à 1
A+
A-
jjwjj22
w
Proximal Support Vector Machines Maximize the Margin between Proximal Planes
x0w = í + 1
x0w = í à 1
A+
A-
jjwjj22
w
Standard Support Vector MachineAlgebra of 2-Category Linearly Separable Case
Given m points in n dimensional space Represented by an m-by-n matrix A Membership of each in class +1 or –1 specified by:A i
An m-by-m diagonal matrix D with +1 & -1 entries
D(Awà eí )=e;
More succinctly:
where e is a vector of ones.
x0w = í æ1: Separate by two bounding planes,
A iw=í + 1; for D i i = + 1;A iw5í à 1; for D i i = à 1:
Standard Support Vector Machine Formulation
Margin is maximized by minimizing21kw;í k2
2
÷> 0 Solve the quadratic program for some :
2÷kyk2
2 + 21kw;í k2
2
D(Awà eí ) + y > ey;w;ímin
s. t.(QP)
,
, denoteswhere D ii = æ1 A+ Aàor membership.
Proximal SVM Formulation (PSVM)
Standard SVM formulation:
w;í (QP)2÷kyk2
2 + 21kw;í k2
2
D(Awà eí ) + y
min
s. t. = e=
This simple, but critical modification, changes the nature of the optimization problem tremendously!!(Regularized Least Squares or Ridge Regression)
Solving for in terms of and gives:
minw;í 2
÷keà D(Awà eí )k22 + 2
1kw; í k22
y w í
Advantages of New Formulation
Objective function remains strongly convex.
An explicit exact solution can be written in terms of the problem data.
PSVM classifier is obtained by solving a single system of linear equations in the usually small dimensional input space.
Exact leave-one-out-correctness can be obtained in terms of problem data.
Linear PSVM
We want to solve:
w;ímin
2÷keà D(Awà eí )k2
2 + 21kw; í k2
2
Setting the gradient equal to zero, gives a nonsingular system of linear equations.
Solution of the system gives the desired PSVM classifier.
Linear PSVM Solution
H = [A à e]Here,
íw
h i= (÷
I + H 0H)à 1H 0De
The linear system to solve depends on:
H 0H(n + 1) â (n + 1)which is of size
is usually much smaller than n m
Linear & Nonlinear PSVM MATLAB Code
function [w, gamma] = psvm(A,d,nu)% PSVM: linear and nonlinear classification% INPUT: A, d=diag(D), nu. OUTPUT: w, gamma% [w, gamma] = psvm(A,d,nu); [m,n]=size(A);e=ones(m,1);H=[A -e]; v=(d’*H)’ %v=H’*D*e; r=(speye(n+1)/nu+H’*H)\v % solve (I/nu+H’*H)r=v w=r(1:n);gamma=r(n+1); % getting w,gamma from r
Numerical experimentsOne-Billion Two-Class Dataset
Synthetic dataset consisting of 1 billion points in 10- dimensional input space Generated by NDC (Normally Distributed Clustered) dataset generatorDataset divided into 500 blocks of 2 million points each.Solution obtained in less than 2 hours and 26 minutes on a 400Mhz About 30% of the time was spent reading data from disk.Testing set Correctness 90.79%
Principal Topics
Knowledge-based classification (NIPS*2002)
Conventional Data-Based SVM
Knowledge-Based SVM via Polyhedral Knowledge Sets
Incoporating Knowledge Sets Into an SVM Classifier
This implication is equivalent to a set of constraints that can be imposed on the classification problem.
Suppose that the knowledge set: belongs to the class A+. Hence it must lie in the halfspace :
èx??Bx 6 b
é
èxjx0w>í + 1
é
Bx6b ) x0w>í + 1
We therefore have the implication:
Numerical TestingThe Promoter Recognition Dataset
Promoter: Short DNA sequence that precedes a gene sequence.
A promoter consists of 57 consecutive DNA nucleotides belonging to {A,G,C,T} .
Important to distinguish between promoters and nonpromoters
This distinction identifies starting locations of genes in long uncharacterized DNA sequences.
The Promoter Recognition DatasetNumerical Representation
Simple “1 of N” mapping scheme for converting nominal attributes into a real valued representation:
Not most economical representation, but commonly used.
The Promoter Recognition DatasetNumerical Representation
Feature space mapped from 57-dimensional nominal space to a real valued 57 x 4=228 dimensional space.
57 nominal values
57 x 4 =228binary values
Promoter Recognition Dataset Prior Knowledge Rules
Prior knowledge consist of the following 64 rules:
R1orR2orR3orR4
2
66666664
3
77777775
V
R5orR6orR7orR8
2
66666664
3
77777775
V
R9or
R10or
R11or
R12
2
66666664
3
77777775
=) PROMOTER
Promoter Recognition Dataset Sample Rules
R8 : (pà 12 = T) ^(pà 11 = A) ^(pà 07 = T);
R4 : (pà 36 = T) ^(pà 35 = T) ^(pà 34 = G)^(pà 33 = A) ^(pà 32 = C);
R10 : (pà 45 = A) ^(pà 44 = A) ^(pà 41 = A);where denotes position of a nucleotide, with respect to a meaningful reference point starting at position and ending at positionpà 50
pj
p7:Then:
R4^R8^R10=) PROMOTER
The Promoter Recognition DatasetComparative Algorithms
KBANN Knowledge-based artificial neural network [Shavlik et al] BP: Standard back propagation for neural networks [Rumelhart et al] O’Neill’s Method Empirical method suggested by biologist O’Neill [O’Neill] NN: Nearest neighbor with k=3 [Cost et al] ID3: Quinlan’s decision tree builder[Quinlan] SVM1: Standard 1-norm SVM [Bradley et al]
The Promoter Recognition DatasetComparative Test Results
Wisconsin Breast Cancer Prognosis Dataset Description of the data
110 instances corresponding to 41 patients whose cancer had recurred and 69 patients whose cancer had not recurred
32 numerical features The domain theory: two simple rules used by doctors:
Wisconsin Breast Cancer Prognosis Dataset Numerical Testing Results
Doctor’s rules applicable to only 32 out of 110 patients.
Only 22 of 32 patients are classified correctly by this rule (20% Correctness).
KSVM linear classifier applicable to all patients with correctness of 66.4%.
Correctness comparable to best available results using conventional SVMs.
KSVM can get classifiers based on knowledge without using any data.
Principal Topics
Fast Newton method classifier
Fast Newton Algorithm for Classification
Standard quadratic programming (QP) formulation of SVM:
Newton Algorithm
f (z) = 21íí (eà D(Awà ew))+
ww2
+ 21íí w; í
íí 2
zi+1 = zi à @2f (zi)à 1r f (zi)Newton algorithm terminates in a finite number of steps
Termination at global minimum
Error rate decreases linearlyCan generate complex nonlinear classifiers
By using nonlinear kernels: K(x,y)
Nonlinear Spiral Dataset94 Red Dots & 94 White Dots
Principal Topics
Breast cancer prognosis & chemotherapy
Kaplan-Meier Curves for Overall Patients:With & Without Chemotherapy
Breast Cancer Prognosis & ChemotherapyGood, Intermediate & Poor Patient Groupings
(6 Input Features : 5 Cytological, 1 Histological)(Grouping: Utilizes 2 Histological Features &Chemotherapy)
Kaplan-Meier Survival Curvesfor Good, Intermediate & Poor Patients
82.7% Classifier Correctness via 3 SVMs
Kaplan-Meier Survival Curves for Intermediate Group Note Reversed Role of Chemotherapy
ConclusionNew methods for classification All based on rigorous mathematical foundationFast computational algorithms capable of classifying massive datasetsClassifiers based on both abstract prior knowledge as well as conventional datasetsIdentification of breast cancer patients that can benefit from chemotherapy
Future WorkExtend proposed methods to broader optimization problems
Linear & quadratic programming Preliminary results beat state-of-the-art software
Incorporate abstract concepts into optimization problems as constraintsDevelop fast online algorithms for intrusion and fraud detection Classify the effectiveness of new drug cocktails in combating various forms of cancer
Encouraging preliminary results for breast cancer
Breast Cancer Treatment ResponseJoint with ExonHit ( French BioTech)
35 patients treated by a drug cocktail 9 partial responders; 26 nonresponders25 gene expression measurements made on each patient1-Norm SVM classifier selected: 12 out of 25 genesCombinatorially selected 6 genes out of 12Separating plane obtained:
2.7915 T11 + 0.13436 S24 -1.0269 U23 -2.8108 Z23 -1.8668 A19 -1.5177 X05 +2899.1 = 0.
Leave-one-out-error: 1 out of 35 (97.1% correctness)
E1 I1 E2 I2 E3 E4 E5I3 I4DNA
3'5'E1 E2 E3I1 I2 E4 E5I3 I4 pre-mRNA
(m=messenger)
Transcription
Alternative RNA splicing
E1 E2 E4 E5E1 E2 E4 E5E3
(A)n(A)nmRNA
Chemo-Sensitive
NH2 COOH
Chemo-Resistant
NH2 COOHProteins
Translation
Detection of Alternative RNA Isoforms via DATAS(Levels of mRNA that Correlate with Senitivity to Chemotherapy)
DATAS
E3
DATAS: Differential Analysis of Transcripts with Alternative Splicing
Talk Available
www.cs.wisc.edu/~olvi