CS480/680: IntrotoMLy328yu/mycourses/480/lec07.pdf · Lecture 07: Hard-margin SVM 1 2020-06-02...
Transcript of CS480/680: IntrotoMLy328yu/mycourses/480/lec07.pdf · Lecture 07: Hard-margin SVM 1 2020-06-02...
CS480/680: Intro to MLLecture 07: Hard-margin SVM
2020-06-02 Yao-Liang Yu1
Outl ine
• Maximum Margin
• Lagrangian Dual
• Alternative View
2020-06-02 Yao-Liang Yu2
Perceptron revisited
2020-06-02 Yao-Liang Yu3
• Two classes: y = 1 or y = -1
• Assuming linearly separable• exist w and b such that for all i,
yi(wTxi + b) > 0
• Find any such w and b feasibility problemmin
w,b0
s.t. 8i, yi(w>xi + b) > 0
Margin
• Take any linear separating hyperplane H
for all i, yi(wTxi + b) > 0
• Move H until it touches some positive point, H1
increase b say
• Move H until it touches some negative point, H-1
decrease b say2020-06-02 Yao-Liang Yu4
margin = dist(H1,H) ∧ dist(H-1,H)
minx,z
kx� zk2 ���� w>
kwk2(x� z)
���2=
|t|kwk2
s.t. w>x+ b = t
w>z+ b = 0
Put into formula
H : wTx + b = 0H1 : wTx + b = tH-1 : wTx + b = -s
What is the distance betweenH1 and H ?
2020-06-02 Yao-Liang Yu5
equality is attainedx = w
kwk22(t� b)
z = wkwk2
2(�b)
Put into formula
H : wTx + b = 0H1 : wTx + b = tH-1 : wTx + b = -t
What is the distance betweenH1 and H ?
2020-06-02 Yao-Liang Yu6
Put into formula
H : wTx + b = 0H1 : wTx + b = 1H-1 : wTx + b = -1
What is the distance betweenH1 and H ?
2020-06-02 Yao-Liang Yu7
Maximum Margin
2020-06-02 Yao-Liang Yu8
maxw,b
1
kwk2s.t. 8i, yi(w>xi + b) � 1
Important facts. • For any f,
• For positive f,
• For s. monotone g, minw
f(w) ⌘ minw
g(f(w))
maxw
1
f(w)=
1
minw f(w)
maxw
f(w) = �minw
�f(w)
Hard-margin Support Vector Machines
2020-06-02 Yao-Liang Yu9
maxw,b
1
kwk2s.t. 8i, yi(w>xi + b) � 1
margin 1
kwk2
Quadratic programming
minw,b
1
2kwk22
s.t. 8i, yi(w>xi + b) � 1
Comparing to perceptron
2020-06-02 Yao-Liang Yu10
minw,b
0
s.t. 8i, yi(w>xi + b) > 0
Linear programming
Support Vectors
• Those touch the parallel hyperplanes H1and H-1
• Usually only a handful
• Entirely determine the hyperplanes!
2020-06-02 Yao-Liang Yu11
Existence and uniqueness
2020-06-02 Yao-Liang Yu12
minw,b
1
2kwk22
s.t. 8i, yi(w>xi + b) � 1
• Always exists a minimizer w and b (if linearly separable)
• The minimizer w is unique (strict convexity of )
• The minimizer b is also unique (why?)
12kwk22
Outl ine
• Maximum Margin
• Lagrangian Dual
• Alternative View
2020-06-02 Yao-Liang Yu13
Lagrangian
2020-06-02 Yao-Liang Yu14
minw,b
max↵�0
1
2kwk22 �
X
i
↵i[yi(w>xi + b)� 1]
Lagrangian
Lagrangian multiplier[dual variable][primal variable]
Primalminw,b
1
2kwk22
s.t. 8i, yi(w>xi + b) � 1
Deriving the dual
2020-06-02 Yao-Liang Yu15
minw,b
max↵�0
1
2kwk22 �
X
i
↵i[yi(w>xi + b)� 1]
max↵�0
minw,b
1
2kwk22 �
X
i
↵i[yi(w>xi + b)� 1]
@
@b=
X
i
↵iyi = 0@
@w= w �
X
i
↵iyixi = 0
The dual problem
2020-06-02 Yao-Liang Yu16
max↵�0
X
i
↵i �1
2
���X
i
↵iyixi
���2
2
s.t.X
i
↵iyi = 0 Only need dot product in the dual !
min↵�0
1
2
X
i
X
j
↵i↵jyiyjx>i xj �
X
k
↵k
s.t.X
i
↵iyi = 0Dual
Rn
Support Vectors
2020-06-02 Yao-Liang Yu17
w =X
i
↵iyixi
↵i > 0
Outl ine
• Maximum Margin
• Dual
• Alternative View
2020-06-02 Yao-Liang Yu18
An dual view
2020-06-02 Yao-Liang Yu19
Convex sets and Convex hul lConvex set. A point set is convex if the line segment [x,y] connecting any two points x and y in C lies entirely in C.
Convex hull. Smallest convex set containing C.
2020-06-02 Yao-Liang Yu20
C 2 Rd
ch(C) :=
(X
i
↵ixi : xi 2 C,↵i � 0,X
i
↵i = 1
).
Separating scale from direct ion
2020-06-02 Yao-Liang Yu21
min↵�0
1
2
���X
i
↵iyixi
���2
2�X
i
↵i
s.t.X
i
↵iyi = 0
minr�0
min↵̄22�
r2
2
���X
i
↵̄iyixi
���2
2� 2r
s.t.X
i
↵̄iyi = 0
<latexit sha1_base64="LJVLRl4GmE/GZSzWmxkW6CvWh+k=">AAACw3icdVHLbtQwFHXCqwyvARYs2FiMQGyIkqgINpWq8lwWiWkrjdPI8dzMWGM7wXagI2P+ic/hW9jgZEYCWriSpXPPffrcqhXc2DT9EcWXLl+5em3n+ujGzVu374zv3jsyTacZTFkjGn1SUQOCK5habgWctBqorAQcV6tXffz4M2jDG/XRrlsoJF0oXnNGbaDK8XciuSqdxmQBn3Dq8Te8YUhFNaGiXVJMuMJEw9zlnrwGYa nHT0JarSlz+jT3gcfkgC/IV2I6WXL8uzY4656Q1C6r2p353hlSy/w0x89wHiaT0RDX0pnEJpvm/2m0h9NyPEmTdDB8EWRbMEFbOyzHP8m8YZ0EZZmgxsyytLWFo9pyJsCPSGegpWxFFzALUFEJpnCDtB4/Dswc140OT1k8sH9WOCqNWcsqZPZ/MOdjPfmv2Kyz9cvCcdV2FhTbDKo7gW2D+zvhOdfArFgHQJnmYVfMljRIbsM1R0TBF9ZISdXc9afxs6xwxMKZHTZ0PeUmmfdBrey8NhfBUZ5ku8nzD7uT/YOtbjvoIXqEnqIMvUD76D06RFPEogfRXvQ2ehe/iVexju0mNY62NffRXxb7X0ub2wU=</latexit>
↵ = r↵̄
<latexit sha1_base64="LTS5UXJY78P1nzhDFYNz4RAXGjI=">AAACJ3icbVBNSxxBFOxR48cadU3IyUvjEshpmZGVeBGW5JKjgawKO8Pypuet29jdM3S/SbI082NyVH9MbmKO+R25pPfjkKgFDdVV7/GKyislHcXxr2hlde3F+sbmVmv75c7uXnv/1bkraytwIEpV2sscHCppcECSFF5WFkHnCi/y648z/+IrWidL84WmFWYarowcSwEUpFH7TQqqmgA/5ZanOdjFd9TuxN14Dv6UJEvSYUucjdp/0qIUtUZDQoFzwySuKPNgSQqFTSutHVYgruEKh4Ea0OgyP4/f8LdBKfi4tOEZ4nP13w0P2rmpzsOkBpq4x95MfM4b1jQ+ybw0VU1oxOLQuFacSj7rghfSoiA1DQSElSErFxOwICg01koNfhOl1mAKn1osmmGS+ZTwO80T+pnkO0nThLaSx908JedH3aTXPf7c6/Q/LHvbZAfskL1jCXvP+uwTO2MDJphnP9gtu4tuop/RffSwGF2Jljuv2X+Ifv8F+mCmkw==</latexit>
r =2
kP
i ↵̄iyixik22
<latexit sha1_base64="DjWuACCqz2igWpudl5dZEnLh7ks=">AAACTHicbVBNa9wwFJS3bZps+rFNj7mILIWcFntJaSkUQnLpMYVuEli55ll+zopIspHkJIvi/5Jf02Pba/9Hb6EQebOHfHRAMJp5D40mr6WwLo7/RL0nT5+tPF9d66+/ePnq9eDNxqGtGsNxwitZmeMcLEqhceKEk3hcGwSVSzzKT/c7/+gMjRWV/ubmNaYKTrQoBQcXpGzwydDPlJUGuB+3nl0y26hMUJaDYSDrGYTLvBMUuFle+os2E+wyG38ft9lgGI/iBehjkizJkCxxkA2uWVHxRqF2XIK10ySuXerBOMEltn3WWKyBn8IJTgPVoNCmfvHHlr4LSkHLyoSjHV2odzc8KGvnKg+TXVT70OvE/3nTxpUfUy903TjU/PahspHUVbQrjBbCIHdyHghwI0JWymcQCnOh1j7TeM4rpUAXnhks2mmSeubwwi0S+k7yw6Tt2koedvOYHI5Hyc7o/ded4e7esrdVskm2yDZJyAeyS76QAzIhnFyRH+QX+R39jP5G19G/29FetNx5S+6ht3IDfDy1Vw==</latexit>
Scale r does not matter
2020-06-02 Yao-Liang Yu22
min↵̄22�
r2
2
���X
i
↵̄iyixi
���2
2� 2r
s.t.X
i
↵̄iyi = 0
<latexit sha1_base64="LkUqqtCsLjlr23w0pCeSwG2dCKQ=">AAADVXicdVJNb9QwEE12Synho1s4csAiLeLCKom2gkulqoDECYpEP6Q4jRzH2Vh1nMh26K6MOfGn+CWIH4OEk13BbgsjWR6/eTNjz3PWMCpVEPx0B8ONW5u3t+54d+/df7A92nl4KutWYHKCa1aL8wxJwignJ4oqRs4bQVCVMXKWXb7u4mefiZC05p/UvCFJhaacFhQjZaF09B1WlKcaZkhAxJoSAUg50JGBbwhTyIBnXwEsBMIalhhxTFgsSJ5ocREZY2kAHtEp/AJlW6UU/C1jD/MOqJAqs0LPTHfoqWl0EYH1Yi9AJGwl6PV0UWk5VuNF6//UPQBBOvKDcdAbuOmES8d3lnac7rjfYF7jtiJcYYakjMOgUYlGQlHMiPFgK0mD8CWakti6HFVEJrqfsQF7FslBUQu7uAI9upqhUSXlvMoss3uEvB7rwH/F4lYVrxJNedMqwvGiUdEyoGrQCQZyKghWbG4dhAW1dwW4RFYRZWX1ICdXuK4qxHMN7TBNHCYaKjJT/Q11B2k/NGuv04vhm9XsP4pESZwxy7NlJFFZPQsOYGk3vetHu2bPg4KhxvZCVJIOHk/2YakCs9rVNrSMlhENr3IbChtlrPn2v3hWt/C6Sjed02gcTsb7Hyf+4dFSwS3nsfPUee6Ezkvn0HnnHDsnDnafuG/d9+6HwY/Br+HGcHNBHbjLnEfOmg23fwP2VRKR</latexit>
P := {i : yi = 1}N := {i : yi = �1}
minµ2�,⌫2�
1
2
���X
i2P
µixi �X
j2N
⌫jxj
���2
2
<latexit sha1_base64="dShlqjHrWB65msjpIEXzRJ4WpgY=">AAADQnicbVFdb9MwFHXC1yiDdfDIi0U2xANUSdQJJECaBg88oSLRbVIcIsd1Wne2E8UOa2XME7+FX8Kf4C/whnjlAaetpnbjSpavzz3Xx74nrzhTOgx/ev616zdu3tq63bmzfffeTnf3/rEqm5rQISl5WZ/mWFHOJB1qpjk9rWqKRc7pSX72pq2ffKa1YqX8qOcVTQUeS1YwgrWDsu53JJjMDBINYhK9pVzjpxDJBl4cLXz8FaIJwZJQntR0lBpU1JiYyJrYWoiO2Bh9QaoRmWGuCw4cJpqMIYH1JC/MzGYMPoNLxrRlvLetRDZdY0xX92TxJ7Mp1mp0sm4Q9sJFwKtJtEoCsIpBtut9Q6OSNIJKTThWKonCSqcG15oRTm0HNYpWmJzhMU1cKrGgKjWLgVq475ARLMraLanhAl3vMFgoNRe5Y7ZfUJdrLfi/WtLo4kVqmKwaTSVZChUNh7qErTtwxGpKNJ+7BJOaubdCMsFu2tp52EGSnpNSCCxHBrnh2CRyZmg604sXmhYyQWQ3fmeWw7Tr3RcTjtMk547nrlFU5+UsfI0mbjN7Qbxn9zuo5rhyWpgp2sK9/gGa6NCuqzpBx2g4Neh85EpRpa2LIF76Fl126WpyHPeifu/gQz84PFo5uAUegkfgCYjAc3AI3oEBGALibXux99J75f/wf/m//T9Lqu+teh6AjfD//gPKPA/h</latexit>
conv({xi : i 2 P})
<latexit sha1_base64="FJmPAEi1Bm1ojfS/T3CbyJS9mFE=">AAACx3icbVFNaxsxEJW3X6n75bTHXkQ3gZSC2TUOLYVCaC/tzYU6CawWo9WOYxGttEhax0boUOgf6K/ptf0Z/TfVrn1wkj4Qerw3o5nRFLXgxibJ31505+69+w/2HvYfPX7y9Nlg//mpUY1mMGVKKH1eUAOCS5habgWc1xpoVQg4Ky4/tf7ZErThSn6z6xryil5IPueM2iDNBm9IRe1CV44pufRHmLhOKOZu5Wf8PeaYcIknmHj8ejaIk2HSAd8m6ZbEaIvJbL/3g5SKNRVIywQ1JkuT2uaOasuZAN8njYGaskt6AVmgklZgctdN5fFhUEo8VzocaXGn7mY4WhmzrooQ2bZsbnqt+D8va+z8Xe64rBsLkm0KzRuBrcLtF+GSa2BWrAOhTPPQK2YLqimz4SP7RMIVU1VFZemIhtJnae6IhZXtOnSt5OLUX5vOMSoZCL+bTRYbMRvlWSFCXHjGgC3UKvlAFuFyB/HowB/2iRa0DrUoN9DKw/ExWdjE71YNBUNEI8CRqzJYaW19QDzyuB/2lt7c0m1yOhqm4+Hx13F88nG7wT30Er1CRyhFb9EJ+owmaIoY+ol+od/oT/QlUtEyWm1Co9425wW6huj7P+S54Fc=</latexit>
conv({xi : i 2 N})
<latexit sha1_base64="/UN5wrAN6tM+iDSmcHAy2eZrM4c=">AAACx3icbVFNaxsxEJW3X6n7Eac99iK6CaQUzK5xaCkUQntpLyWFOgmsFqPVjrMiWmmRtI6N0KHQP9Bf02v7M/pvqrV9cJI+EHq8N6OZ0RSN4MYmyd9edOfuvfsPdh72Hz1+8nR3sPfs1KhWM5gwJZQ+L6gBwSVMLLcCzhsNtC4EnBWXHzv/bA7acCW/2WUDeU0vJJ9xRm2QpoPXpKa20rVjSs79ISZuJRQzt/BT/g5zTLjEXzDx+NV0ECfDZAV8m6QbEqMNTqZ7vR+kVKytQVomqDFZmjQ2d1RbzgT4PmkNNJRd0gvIApW0BpO71VQeHwSlxDOlw5EWr9TtDEdrY5Z1ESK7ls1NrxP/52Wtnb3NHZdNa0GydaFZK7BVuPsiXHINzIplIJRpHnrFrKKaMhs+sk8kXDFV11SWjmgofZbmjlhY2FWHrpNcnPpr0zlGJQPht7NJtRazUZ4VIsSFZwzYQi2S96QKl9uPR/v+oE+0oE2oRbmBTh6Oj0hlE79dNRQMEa0AR67KYKWN9QHxyON+2Ft6c0u3yelomI6HR1/H8fGHzQZ30Av0Eh2iFL1Bx+gTOkETxNBP9Av9Rn+iz5GK5tFiHRr1NjnP0TVE3/8B4EHgVQ==</latexit>
↵̄ = [µ; ⌫]
<latexit sha1_base64="ltYFHPY1fPGLbyrDxAawpT+m2+c=">AAACAHicbVBNS8NAEN34WetX1IMHL4tF8FQSqSiIUPTisYL9gCSUyXbTLt1swu5GKKEX/4oXD4p49Wd489+4bXPQ1gcDj/dmmJkXppwp7Tjf1tLyyuraemmjvLm1vbNr7+23VJJJQpsk4YnshKAoZ4I2NdOcdlJJIQ45bYfD24nffqRSsUQ86FFKgxj6gkWMgDZS1z70Q5A+8HQA+Bp7fpxdYV9kQdeuOFVnCrxI3IJUUIFG1/7yewnJYio04aCU5zqpDnKQmhFOx2U/UzQFMoQ+9QwVEFMV5NMHxvjEKD0cJdKU0Hiq/p7IIVZqFIemMwY9UPPeRPzP8zIdXQY5E2mmqSCzRVHGsU7wJA3cY5ISzUeGAJHM3IrJACQQbTIrmxDc+ZcXSeus6taq5/e1Sv2miKOEjtAxOkUuukB1dIcaqIkIGqNn9IrerCfrxXq3PmatS1Yxc4D+wPr8AVP9lZk=</latexit>
NOW this
2020-06-02 Yao-Liang Yu23
w = rX
i
↵̄iyixi
/X
i2P
µixi �X
i2N
⌫ixi
<latexit sha1_base64="tbUZ/m05DO1k/knSQ19TNgc2lec=">AAACb3icbZHPSisxFMYz4/9er1ZduFAuwaJcF5YZUXQjiG5cSQWrQlPKmTRjg5lMSDJqGWbrA7rzHdz4BmbaWWj1QODLd36HJF8iJbixQfDm+VPTM7Nz8wu1P4t/l5brK6s3Js00ZW2ailTfRWCY4JK1LbeC3SnNIIkEu40ezsv+7SPThqfy2g4V6yZwL3nMKVhn9eovJAE7iOL8qcA7J1hjYrKkxzGJQBMQagBuMyyNinsuyg2p7WCidKpsiscjOSdc4lbhwGwS3/uKXDpETiK9eiNoBqPCP0VYiQaqqtWrv5J+SrOESUsFGNMJA2W7OWjLqWBFjWSGKaAPcM86TkpImOnmo7wKvO2cPo5T7Za0eOR+ncghMWaYRI4s72gme6X5W6+T2fi4m3OpMsskHR8UZwK7lMrwcZ9rRq0YOgFUc3dXTAeggVr3RTUXQjj55J/iZr8ZHjQPrw4ap2dVHPNoA22h/yhER+gUXaAWaiOK3r1Vb8Pb9D78df+fj8eo71Uza+hb+buf5hq71w==</latexit>
r =2
kP
i ↵̄iyixik22
<latexit sha1_base64="DjWuACCqz2igWpudl5dZEnLh7ks=">AAACTHicbVBNa9wwFJS3bZps+rFNj7mILIWcFntJaSkUQnLpMYVuEli55ll+zopIspHkJIvi/5Jf02Pba/9Hb6EQebOHfHRAMJp5D40mr6WwLo7/RL0nT5+tPF9d66+/ePnq9eDNxqGtGsNxwitZmeMcLEqhceKEk3hcGwSVSzzKT/c7/+gMjRWV/ubmNaYKTrQoBQcXpGzwydDPlJUGuB+3nl0y26hMUJaDYSDrGYTLvBMUuFle+os2E+wyG38ft9lgGI/iBehjkizJkCxxkA2uWVHxRqF2XIK10ySuXerBOMEltn3WWKyBn8IJTgPVoNCmfvHHlr4LSkHLyoSjHV2odzc8KGvnKg+TXVT70OvE/3nTxpUfUy903TjU/PahspHUVbQrjBbCIHdyHghwI0JWymcQCnOh1j7TeM4rpUAXnhks2mmSeubwwi0S+k7yw6Tt2koedvOYHI5Hyc7o/ded4e7esrdVskm2yDZJyAeyS76QAzIhnFyRH+QX+R39jP5G19G/29FetNx5S+6ht3IDfDy1Vw==</latexit>
Questions?
2020-06-02 Yao-Liang Yu24