Announcements Homework 4 is due on this Thursday (02/27/2004) Project proposal is due on 03/02.
-
date post
20-Dec-2015 -
Category
Documents
-
view
223 -
download
5
Transcript of Announcements Homework 4 is due on this Thursday (02/27/2004) Project proposal is due on 03/02.
Announcements Homework 4 is due on this Thursday (02/27/2004) Project proposal is due on 03/02
Unconstrained Optimization
Rong Jin
Logistic Regression2
2 1 1
1( ) ( ) log
1 exp ( )N m
reg train train ii il D l D w s w
y b x w
The optimization problem is to find weights w and b that maximizes the above log-likelihood
How to do it efficiently ?
Gradient Ascent Compute the gradient
Increase weights w and threshold b in the gradient direction
21 1
21 1
log ( | )
log ( | )
where is learning rate.
N mi i ii i
N mi i ii i
w w p y x s ww
c c p y x s wc
21 1 1
21 1 1
log ( | ) (1 ( | ))
log ( | ) (1 ( | ))
N m Ni i i i i i ii i i
N m Ni i i i i ii i i
p y x s w sw x y p y xw
p y x s w y p y xb
Problem with Gradient Ascent Difficult to find the appropriate step
size Small slow convergence Large oscillation or “bubbling”
Convergence conditions Robbins-Monroe conditions
Along with “regular” objective function will ensure convergence
20 0
, t tt t
Newton Method Utilizing the second order derivative Expand the objective function to the second order around x0
The minimum point is Newton method for optimization
Guarantee to converge when the objective function is convex
0 0
20 0 0( ) ( ) ( ) ( )
2'( ) , ''( )
x x x x
bf x f x a x x x x
a f x b f x
0 /x x a b
'( )
''( )
old
old
new old x x
x x
f xx x
f x
Multivariate Newton Method Object function comprises of multiple variables
Example: logistic regression model
Text categorization: thousands of words thousands of variables Multivariate Newton Method
Multivariate function:
First order derivative a vector
Second order derivative Hessian matrix Hessian matrix is mxm matrix Each element in Hessian matrix is defined as:
21 1
1( ) log
1 exp ( )N m
reg train ii il D s w
y b x w
21 2
,( , ,..., )m
i ji j
f x x x
x x
H
1 2( ) ( , ,..., )mf x f x x x
1 1, ,...,
m
f f f f
x x x x
Multivariate Newton Method Updating equation:
Hessian matrix for logistic regression model
Can be expensive to compute Example: text categorization with 10,000 words Hessian matrix is of size 10,000 x 10,000 100 million entries Even worse, we have compute the inverse of Hessian matrix H-1
1 ( )new old f xx x
x
H
1( | )(1 ( | ))
n Ti i i i m mi
p y x p y x x x s H I
Quasi-Newton Method Approximate the Hessian matrix H with another B matrix:
B is update iteratively (BFGS):
Utilizing derivatives of previous iterations
1 ( )new old f xx x
x
B
1
1 1,
T Tk k k k k k
k k T Tk k k k k
k k k k k k
p p y y
p B p y p
p x x y g g
B BB B
Limited-Memory Quasi-Newton Quasi-Newton
Avoid computing the inverse of Hessian matrix But, it still requires computing the B matrix large storage
Limited-Memory Quasi-Newton (L-BFGS) Even avoid explicitly computing B matrix
B can be expressed as a product of vectors Only keep the most recently vectors of (3~20)
1
1 1,
T Tk k k k k k
k k T Tk k k k k
k k k k k k
p p y y
p B p y p
p x x y g g
B BB B
{ , }k kp y
{ , }k kp y
Linear Conjugate Gradient Method Consider optimizing the quadratic function
Conjugate vectors The set of vector {p1, p2, …, pl} is said to be conjugate with respect to
a matrix A if
Important property The quadratic function can be optimized by simply optimizing the
function along individual direction in the conjugate set. Optimal solution:
k is the minimizer along the kth conjugate direction
* arg min2
TT
x
x xx b x
A
0, for any Ti jp p i j A
1 1 2 2 ... l lx p p p
Example Minimize the following function
Matrix A
Conjugate direction
Optimization First direction, x1 = x2=x:
Second direction, x1 =- x2=x:
Solution: x1 = x2=1
-4
-2
0
2
4
-3
-2
-1
0
1
2
3-10
0
10
20
302 21 2 1 2 1 2 1 2( , )f x x x x x x x x
1 0.5
0.5 1A
1 21 1
,1 1
p p
21 2 1( , ) 2 Minimizer 1f x x x x
21 2 2( , ) 2 Minimizer 0f x x x
How to Efficiently Find a Set of Conjugate Directions Iterative procedure
Given conjugate directions {p1,p2,…, pk-1}
Set pk as follows:
Theorem: The direction generated in the above step is conjugate to all previous directions {p1,p2,…, pk-1}, i.e.,
Note: compute the k direction pk only requires the previous direction pk-1
11
1 1
( ), , where
k
Tk k
k k k k k kTx xk k
r Ap f xp r p r
xp Ap
, for any [1, 2,..., 1]Tk ip p i k A
Nonlinear Conjugate Gradient Even though conjugate gradient is derived for a
quadratic objective function, it can be applied directly to other nonlinear functions
Several variants: Fletcher-Reeves conjugate gradient (FR-CG) Polak-Ribiere conjugate gradient (PR-CG)
More robust than FR-CG
Compared to Newton method No need for computing the Hessian matrix No need for storing the Hessian matrix
Generalizing Decision Trees
+ +
a decision tree with simple data partition
+
a decision tree using classifiers for data partition
+
Each node is a linear classifier
Attribute 1
Attribute 2
classifier
Generalized Decision Trees Each node is a linear classifier Pro:
Usually result in shallow trees Introducing nonlinearity into linear classifiers (e.g. logistic
regression) Overcoming overfitting issues through the regularization mechanism
within the classifier. Better way to deal with real-value attributes
Example: Neural network Hierarchical Mixture Expert Model
Example
Kernel method
x=0
x=0
x=0
Generalized Tree
+ +
Hierarchical Mixture Expert Model (HME)
Group 1
g1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
Group 2
g2(x)
m1,2(x) m2,1(x) m2,2(x)
X
y
• Ask r(x): which group should be used for classifying input x ?
• If group 1 is chosen, which classifier m(x) should be used ?
• Classify input x using the chosen classifier m(x)
Hierarchical Mixture Expert Model (HME)Probabilistic Description
Group 1
g1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
Group 2
g2(x)
m1,2(x) m2,1(x) m2,2(x)
X
y
1 11
1 12
2 21
2 22
( | ) ( , , | )
( 1 | ) ( | )( 1| )
( 1| ) ( | )
( 1| ) ( | )( 1| )
( 1| ) ( | )
g mp y x p y g m x
g x m y xr x
g x m y x
g x m y xr x
g x m y x
Two hidden variables
The hidden variable for groups:g = {1, 2}
The hidden variable for classifiers:m = {11, 12, 21, 22}
Hierarchical Mixture Expert Model (HME)Example
Group 1
g1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
Group 2
g2(x)
m1,2(x) m2,1(x) m2,2(x)
X
y
r(+1|x) = ¾, r(-1|x) = ¼
g1(+1|x) = ¼, g1(-1|x) = ¾
g2(+1|x) = ½ , g2(-1|x) = ½
+1 -1
m1,1(x) ¼ ¾
m1,2(x) ¾ ¼
m2,1(x) ¼ ¾
m2,2(x) ¾ ¼
¾ ¼
¼ ¾ ½ ½
p(+1|x) = ?, p(-1|x) = ?
Training HME In the training examples {xi, yi}
No information about r(x), g(x) for each example Random variables g, m are called hidden variables since
they are not exposed in the training data. How to train a model with hidden variable?
Start with Random Guess …
xg1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
g2(x)
m1,2(x) m2,1(x) m2,2(x)
+: {1, 2, 3, 4, 5}
: {6, 7, 8, 9}
Randomly Assignment
• Randomly assign points to each group and expert
• Learn classifiers r(x), g(x), m(x) using the randomly assigned points
{1,2,} {6,7} {3,4,5} {8,9}
{1}{6} {2}{7} {3}{9} {5,4}{8}
Adjust Group Memeberships
xg1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
g2(x)
m1,2(x) m2,1(x) m2,2(x)
+: {1, 2, 3, 4, 5}
: {6, 7, 8, 9}
• The key is to assign each data point to the group who classifies the data point correctly with the largest probability
• How ?{1,2} {6,7} {3,4,5} {8,9}
{1}{6} {2}{7} {3}{9} {5,4}{8}
Adjust Group Memberships
xg1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
g2(x)
m1,2(x) m2,1(x) m2,2(x)
+: {1, 2, 3, 4, 5}
: {6, 7, 8, 9}
• The key is to assign each data point to the group who classifies the data point correctly with the largest confidence
• Compute p(g=1|x,y) and p(g=2|x,y){1,2} {6,7} {3,4,5} {8,9}
{1}{6} {2}{7} {3}{9} {5,4}{8}
Posterior Prob. For Groups
Group 1 Group 2
1 0.8 0.2
2 0.4 0.6
3 0.3 0.7
4 0.1 0.9
5 0.65 0.35
Adjust Memberships for Classifiers
xg1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
g2(x)
m1,2(x) m2,1(x) m2,2(x)
+: {1, 2, 3, 4, 5}
: {6, 7, 8, 9}
{1,5} {6,7} {2,3,4} {8,9}
• The key is to assign each data point to the classifier who classifies the data point correctly with the largest confidence
• Compute p(m=1,1|x, y), p(m=1,2|x, y), p(m=2,1|x, y), p(m=2,2|x, y)
Adjust Memberships for Classifiers
xg1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
g2(x)
m1,2(x) m2,1(x) m2,2(x)
+: {1, 2, 3, 4, 5}
: {6, 7, 8, 9}
{1,5} {6,7} {2,3,4} {8,9}
Posterior Prob. For Classifiers
1 2 3 4 5
m1,10.7 0.1 0.15 0.1 0.05
m1,20.2 0.2 0.20 0.1 0.55
m2,10.05 0.5 0.60 0.1 0.3
m2,10.05 0.2 0.05 0.7 0.1
• The key is to assign each data point to the classifier who classifies the data point correctly with the largest confidence
• Compute p(m=1,1|x, y), p(m=1,2|x, y), p(m=2,1|x, y), p(m=2,2|x, y)
Adjust Memberships for Classifiers
xg1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
g2(x)
m1,2(x) m2,1(x) m2,2(x)
+: {1, 2, 3, 4, 5}
: {6, 7, 8, 9}
{1,5} {6,7} {2,3,4} {8,9}
Posterior Prob. For Classifiers
1 2 3 4 5
m1,10.7 0.1 0.15 0.1 0.05
m1,20.2 0.2 0.20 0.1 0.55
m2,10.05 0.5 0.60 0.1 0.3
m2,10.05 0.2 0.05 0.7 0.1
• The key is to assign each data point to the classifier who classifies the data point correctly with the largest confidence
• Compute p(m=1,1|x, y), p(m=1,2|x, y), p(m=2,1|x, y), p(m=2,2|x, y)
Adjust Memberships for Classifiers
Posterior Prob. For Classifiers
1 2 3 4 5
m1,10.7 0.1 0.15 0.1 0.05
m1,20.2 0.2 0.20 0.1 0.55
m2,10.05 0.5 0.60 0.1 0.3
m2,10.05 0.2 0.05 0.7 0.1
• The key is to assign each data point to the classifier who classifies the data point correctly with the largest confidence
• Compute p(m=1,1|x, y), p(m=1,2|x, y), p(m=2,1|x, y), p(m=2,2|x, y)
xg1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
g2(x)
m1,2(x) m2,1(x) m2,2(x)
+: {1, 2, 3, 4, 5}
: {6, 7, 8, 9}
{1,5} {6,7} {2,3,4} {8,9}
{1}{6} {5}{7} {2,3}{9} {4}{8}
Retrain The Model
• Retrain r(x), g(x), m(x) using the new memberships
xg1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
g2(x)
m1,2(x) m2,1(x) m2,2(x)
+: {1, 2, 3, 4, 5}
: {6, 7, 8, 9}
{1,5} {6,7} {2,3,4} {8,9}
{1}{6} {5}{7} {2,3}{9} {4}{8}
Expectation Maximization Two things need to estimate
Logistic regression models for r(x;r), g(x; g) and m(x;m)
Unknown group memberships and expert memberships
p(g=1,2|x), p(m=11,12|x,g=1), p(m=21,22|x,g=2)
E-step
1. Estimate p(g=1|x, y), p(g=2|x, y) for training examples, given guessed r(x;r), g(x;g) and m(x;m)
2. Estimate p(m=11, 12|x, y) and p(m=21, 22|x, y) for all training examples, given guessed r(x;r), g(x;g) and m(x;m)
M-step
1. Train r(x;r) using weighted examples: for each x, p(g=1|x) fraction as a positive example, and p(g=2|x) fraction as a negative example
2. Train g1(x; g) using weighted examples: for each x, p(g=1|x)p(m=11|x,g=1) fraction as a positive example and p(g=1|x)p(m=12|x,g=1) fraction as a negative example. Training g2(x; g) similarly
3. Train m(x;m) with appropriately weighted examples
Comparison of Different Classification Models The goal of all classifiers
Predicating class label y for an input x Estimate p(y|x)
Gaussian generative model p(y|x) ~ p(x|y) p(y): posterior = likelihood prior p(x|y)
Describing the input patterns for each class y Difficult to estimate if x is of high dimensionality Naïve Bayes: p(x|y) ~ p(x1|y) p(x2|y)… p(xm|y)
Essentially a linear model Linear discriminative model
Directly estimate p(y|x) Focusing on finding the decision boundary
Comparison of Different Classification Models Logistic regression model
A linear decision boundary: wx+b
A probabilistic model p(y|x)
Maximum likelihood approach for estimating weights w and threshold b
0 positive
0 negative
w x b
w x b
1( 1| )
1 exp( ( ))p y x
y w x b
( ) ( )
1 1
( ) ( )
1 1
( ) log ( | ) log ( | )
1 1log log
1 exp 1 exp
N Ntrain i ii i
N N
i ii i
l D p x p x
w x b w x b
1w x b
Comparison of Different Classification Models Logistic regression model
Overfitting issue Example: text classification
Every word is assigned with a different weight Words that appears in only one document will be assigned with infinite
large weight Solution: regularization
( ) ( )
21 1
( ) ( ) 21 1 1
( ) log ( | ) log ( | )
1 1log log
1 exp 1 exp
N Ntrain i ii i
N N mji i j
i i
l D p x p x s w
s ww x b w x b
Regularization term
Comparison of Different Classification Models Conditional exponential model
An extension of logistic regression model to multiple class case
A different set of weights wy and threshold b for each class y
Maximum entropy model Finding the simplest model that matches with the data
1( | ; ) exp( )
( )
( ) exp( )
y y
y yy
p y x c x wZ x
Z x c x w
1 1( | )
1 1
max ( | ) max ( | ) log ( | )
subject to
( | ) ( , ), ( | )=1
i
N Ni i i ii i yp y x p
N Ni i i i ii i y
H y x p y x p y x
p y x x x y y p y x
Maximize Entropy Prefer uniform distribution
Constraints Enforce the model to be consistent
with observed data
Classification Margin
Comparison of Different Classification Models
Support vector machine Classification margin Maximum margin principle:
Separate data far away from the decision boundary
Two objectives Minimize the classification
error over training data Maximize the classification
margin Support vector
Only support vectors have impact on the location of decision boundary
denotes +1
denotes -1
Support Vectors
0w x b
Comparison of Different Classification Models Separable case
Noisy case
* * 21
,
1 1
2 2
{ , }= argmin
subject to
1
1
....
1
mii
w b
N N
w b w
y w x b
y w x b
y w x b
* * 21 1
,
1 1 1 1
2 2 2 2
{ , }= argmin
subject to
1 , 0
1 , 0
....
1 , 0
m Ni ji j
w b
N N N N
w b w c
y w x b
y w x b
y w x b
Quadratic programming!
Comparison of Classification Models Logistic regression model vs. support vector machine
21 1
,
1{ , }* arg max log
1 exp ( )
N mji j
w b i
w b s wy w x b
* * 21 1
,
1 1 1 1
{ , }= argmin
subject to
1 , 0
....
1 , 0
N mi ji j
w b
N N N N
w b c w
y w x b
y w x b
Log-likelihood can be viewed as a measurement
of accuracy
Identical terms
Comparison of Different Classification Models
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 10
0.5
1
1.5
2
2.5
3
3.5
wx+b
Loss
Loss function for logistic regressionLoss function for SVM
Logistic regression differs from support vector machine only in the loss function
0.5 1 1.5 2 2.5 3 3.5 4 4.50
10
20
30
40
50
60
X
Cou
nt
fitting curve for positive datafitting curve for negative datahistogram for negative datahistogram for positive data
Comparison of Different Classification Models
• Generative models have trouble at the decision boundary
• Classification boundary that achieves the least training error
• Classification boundary that achieves large margin
Nonlinear Models Kernel methods
Add additional dimensions to help separate data
Efficiently computing the dot product in a high dimension space
Kernel method
x=0
x=0
( ) ( ) ( , )w x K w x Φ Φ
( )x x Φ
Nonlinear Models
Decision trees Nonlinearly combine different
features through a tree structure
Hierarchical Mixture Model Replace each node with a
logistic regression model Nonlinearly combine multiple
linear models
+ + +
Group 1
g1(x)
m1,1(x)
Group Layer
ExpertLayer
r(x)
Group 2
g2(x)
m1,2(x) m2,1(x) m2,2(x)