Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke...
Transcript of Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke...
![Page 1: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/1.jpg)
Statistical Learning for Image Orientation
Luke Barrington
Mentors: Nuno Vasconcelos, UCSD ECE Dept.Babak Jafarian, Cal-(IT)2
![Page 2: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/2.jpg)
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
![Page 3: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/3.jpg)
Project OverviewDevelop an image classification system for MMS-enabled cell phonesMotivation is for use by spectators at Athens 2004 Olympic gamesUse modern classification technologies to improve server-side organization and provide useful user output
Goal for my project: image orientation
![Page 4: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/4.jpg)
Sports Photographers
![Page 5: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/5.jpg)
Image OrientationCamera phones, scanners, digital cameras, online image databases, …Device output has standard dimension
At least 50% will be incorrect
Bulk, automated, corrective processing before display to human users
![Page 6: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/6.jpg)
System Block Diagram
MMS/Internet server
.
.
.
.
.
.
UnorderedDirectory
OrderedOutputUser Input
Classifier
![Page 7: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/7.jpg)
Classifier Block DiagramUser’s Parameters
+1-1…+1
KernelErrorWeights
TRAIN ModelParameters CLASSIFY
+1
-1
Training DataTraining Data
Data Labels
Class Label
Unlabeled Data
![Page 8: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/8.jpg)
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
![Page 9: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/9.jpg)
Decision Boundary
Margin
Training = Learn Decision Boundary
-1 training examples
+1training examples
Support Vectors
![Page 10: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/10.jpg)
Decision Boundary
Classification
Support Vectors
-1
+1
B
Test B = +1
A
Test A = -1
![Page 11: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/11.jpg)
Extension to Multi-Class Problems
SVM is a binary decision classifierOne solution is a decision tree format
n-class problem requires ½n(n-1) classifiers
A B C D
A/B C/D
A/C A/D B/C B/D
![Page 12: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/12.jpg)
Multi-Class: 1-vs-All ClassifierAlternative is a 1-vs-all classifierTrain each class with all training dataLabel: +1 (in the class) or
–1 (in any other class)Requires n classifiers for an n-class problemOutput may be undetermined (all classifiers return –1) Class +1 -1
a a {b, …, n}b b {a, c, …, n}
n n {a, …, n-1}M MM
![Page 13: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/13.jpg)
Linear Support Vector Machine for separable dataAssume the training points can be divided by a linear hyperplane:
xi●w + b = 0 w is the normal vector
is perpendicular distance to origin
With appropriate scaling, the training data satisfy:xi●w + b ≥ +1 for yi = +1xi●w + b ≤ -1 for yi = -1
Combine this into a single condition: yi(xi●w + b) – 1 ≥ 0 for all i
The width of margin is: 2/||w||Robust classifier has widest margin => minimize ||w||2
wb−
![Page 14: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/14.jpg)
Linear SVM, separable data contd.
Primal Lagrangian formulation:
Setting the gradient of LP w.r.t. w & b = 0 gives the conditions:
Combining these equality constraints in the dual formulation gives:
SVM training now amounts to maximizing LD with respect to the weights αi, subject to the constraints above and αi ≥ 0
Support vectors are all the training points where αi > 0
For classifier’s purposes, all other points can be ignored
∑∑ ⋅−== ji
jijiji
l
iiD xxyyL
,1
ααα
( ) ∑∑==
++⋅−=l
ii
l
iiiiP bwxywL
11
2
21 αα
011
== ∑∑==
l
iii
l
iiii yandxyw αα
![Page 15: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/15.jpg)
Linear SVM for unseparable dataWe modify the equations from linear and separable case by the addition of a “slack” variable ξ. Now the equations become
xi●w + b ≥ +1 – ξi for yi = +1xi●w + b ≤ -1 + ξi for yi = -1 ξi ≥ 0if ξi > 1 => an error occurred Σiξi is an upper-bound on the training error
Now we want to minimize:
C is a user-specified cost parameter
This is again a constrained optimization problem where we want to maximize:
Subject to:
The data which satisfy this maximization are the Support Vectors
∑∑ ⋅−== ji
jijiji
l
iiD xxyyL
,1ααα
k
iiC
w⎟⎠
⎞⎜⎝
⎛− ∑ξ
2
2
001
=≤≤ ∑=
l
iiii yandC αα
![Page 16: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/16.jpg)
Non-linear SVMsGeneralize to non-linear decision function for the classifierMap the data to some other (higher-dimension) Euclidean space H:
Now the problem depends only on
Find a kernel function such that
Use this (non-infinite) result to get classifier output
HRd →Φ :
)()( ji xx Φ⋅Φ
)()(),( jiji xxxxK Φ⋅Φ=
( ) ∑∑==
+=+Φ⋅Φ=SS N
iiii
N
iiii bxsKybxsyxf
11),()()( αα
![Page 17: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/17.jpg)
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
![Page 18: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/18.jpg)
SVM ParametersKernel:
Linear (dot product) K(xi,xj) = (xi.xj)Polynomial: K(xi,xj) = (xi.xj + 1)p
RBF: K(xi,xj) = e-|| xi-xj ||²/2σ²
Sigmoid neural network: K(xi,xj) = tanh(κxi.xj -δ)
C (Error penalty) :Sets an upper bound on the alpha coefficients Higher C gives higher weight to outliersLow C assumes many data points are outliersAssign different C for each class based on number training data points
![Page 19: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/19.jpg)
Parameter Selection“Research” = correct choice of parametersRadial Basis Function generalizes bestParameter space is 2-DFind optimum set using grid search
σ
C
![Page 20: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/20.jpg)
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
![Page 21: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/21.jpg)
Sample Problem: Car Orientation
= +1
= -1
![Page 22: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/22.jpg)
Feature Space
Grayscale pixel values (0 to 255)No input scalingDown-sample image to 128x128 and row scan
Input data is 16384-dimensional vector
![Page 23: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/23.jpg)
Data SetFrom MIT labs516 imagesTraining set: 400Validation set: 50RBF kernel, σ = 4 C = 25
![Page 24: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/24.jpg)
Car Orientation ResultsClassifiers for {0º, 30º, 60º, 90º, 120º, 150º, 180º}
In-class detection accuracy = 97.5 ± 2%False Positive < 2%
![Page 25: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/25.jpg)
Initial ResultsSVM solves this problem easily on a limited domainTraining time can be significant (large feature size)Classification is almost instantaneous
![Page 26: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/26.jpg)
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
![Page 27: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/27.jpg)
Expansion of Training SetEfficacy of classifier depends on training data
Corel Database
Sports Images
Crawl web sites
Use subclasses to choose feature spaceTrain “final” classifier on everything
![Page 28: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/28.jpg)
Feature SpaceGrayscale pixel valuesSquare images
Each image provides 4 training examples
Down-sample image to 112x112Input data is 12544 dimensional vector
= +1 = -1
![Page 29: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/29.jpg)
ResultsBest Corel class
70% detection rate, 3% false positiveWorst Corel class
0% detection rate, 30% false positiveSports images
63% detection rate, 14% false positive
Still only working on one specific class at a time
![Page 30: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/30.jpg)
New Feature SpacePixel data is over-sized and under-poweredNeed to capture edge information in more succinct and compact formatEdge orientation histogram
Eigenvalues of gradient matrix
![Page 31: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/31.jpg)
Gradient Matrix
comes from edge detection
⎟⎟⎠
⎞⎜⎜⎝
⎛=0k
⎟⎟⎠
⎞⎜⎜⎝
⎛=
k0
for vertical edges for horizontal edges
•Construct a gradient field on N x N image sub-blocks
•Sub-block size depends on image size
•Eigenvectors of H are features
•Use N = 16, 8, 4, 2, 1 to capture both local and global features
•Feature size = 2(162 + 82 + 42 + 22 + 1) = 682
![Page 32: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/32.jpg)
Using Color DataPixel intensity alone requires 12544 dimensional input spaceAdding color => increase by 200%HSV space mean & varianceat image borders (Wang & Zhang 2003)
![Page 33: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/33.jpg)
Current WorkApply new feature space to experiments reviewedExplore color feature representationsTrain general system across all classesOptimize system to work with camera phone outputDevelop web site / application for digital photos
Add external modules (e.g. face detection, scene classification) to improve performance
![Page 34: Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke Barrington Mentors: Nuno Vasconcelos, UCSD ECE Dept. Babak Jafarian, Cal-(IT)2. Agenda](https://reader036.fdocuments.net/reader036/viewer/2022070803/5f032def7e708231d407eed8/html5/thumbnails/34.jpg)
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions