Post on 14-Mar-2021
CSE586 Robert Collins
CSE 586, Spring 2010 Advanced Computer Vision
Procrustes Shape Analysis PART 2: Active Shape Models
CSE586 Robert Collins
Lecture material from Tim Cootes University of Manchester. For more info, see http://www.isbe.man.ac.uk/~bim/ (includes code for exploring active shape/appearance models).
Credits
lots of slides are due to
CSE586 Robert Collins
Generalized Procrustes Analysis
CSE586 Robert Collins
Shape Space
Consider K landmark points (xi,yi) in 2D
Dimension of the space of observations is 2*K
But what is dimension of the space of the shapes?
Recall, two point configurations are equivalent (have the same shape) if one can be translated, rotated and scaled into the other.
Thus, each unique “shape” is an equivalence class.
CSE586 Robert Collins
Shape Space
The shape space is then the quotient space induced by this equivalence class on the observation space
So shape space has dimension 2*K - 4
This is often called Kendall’s Shape Space.
k
rotation translation scaling
CSE586 Robert Collins
Shape Space
For triangles (K=3), Kendall’s shape space can be visualized as a sphere.
For more points, the space is a more complicated manifold (is it a hypersphere?)
Rather than measure variation in shape on the sphere (or higher dimensional nonlinear manifold), it is common to project points into the tangent plane taken at the mean shape. This is a linear space, so can use PCA, Gaussians, mixture of Gaussians, etc.
CSE586 Robert Collins
Shape Space
http://www.york.ac.uk/res/fme/resources/morphologika/helpfiles/shapespace.html
CSE586 Robert Collins
Shape Space
Tangent projection can be carried out by subtracting the vector of normalized shape coordinates of a figure from the mean vector, and taking just the component perpendicular to the mean.
tangent plane
mean shape
nonlinear shape space
shape sample
shape sample projected into tangent plane
CSE586 Robert Collins
Shape Space
If shape variation is relatively small, we can just take the shape difference vector as an approximation to the vector in the tangent plane.
tangent plane
mean shape
nonlinear shape space
shape sample
CSE586 Robert Collins
Aligned Shapes
• Note: need to model the aligned shapes
x
space shape
We talked about this two lectures ago
CSE586 Robert Collins
Statistical Shape Models
• For shape synthesis – Parameterised model preferable
• For image matching we can get away with only knowing p(b) – Usually more efficient to reduce dimensionality
where possible
)(bx shapef= Pbxx += e.g.
CSE586 Robert Collins
Dimensionality Reduction
• Coordinates often correlated • Nearby points move together
11bpxx +≈
1b
xx
1p
CSE586 Robert Collins
Principal Component Analysis
• Compute eigenvectors of covariance,S • Eigenvectors : main directions • Eigenvalue : variance along eigenvector
1p2p
1λ∝ 2λ∝We talked about this last time
CSE586 Robert Collins
Dimensionality Reduction
• Data lies in subspace of reduced dim.
• However, for some t,
iλ
i
tjbj >≈ if 0
t
) is of (Variance jjb λ
nnbb ppxx +++= L11 ...
CSE586 Robert Collins
Building Shape Models
• Given aligned shapes, { } • Apply PCA
• P – First t eigenvectors of covar. matrix • b – Shape model parameters
ix
Pbxx +≈
CSE586 Robert Collins
Hand shape model
• 72 points placed around boundary of hand – 18 hand outlines obtained by thresholding images of
hand on a white background
• Primary landmarks chosen at tips of fingers and joint between fingers – Other points placed equally between
1 2 3
4
5 6
CSE586 Robert Collins
Labeling Examples
From Stegmann and Gomez
CSE586 Robert Collins
Labeling Examples
From Stegmann and Gomez
CSE586 Robert Collins
Point Alignment
From Stegmann and Gomez
Training points before and after alignment
CSE586 Robert Collins
Hand Shape Model
Samples from training set, after alignment
CSE586 Robert Collins
Component Analysis
From Stegmann and Gomez
Full hand shape model is built by computing PCA on all model points collectively, which captures correlations of motion between points.
Independent PCA on each data point cluster
CSE586 Robert Collins
Hand Shape Model
Varying b1
Varying b2
Varying b3
First 3 modes of variation
CSE586 Robert Collins
Face Shape Example
Sample face training image
1st mode of variation
CSE586 Robert Collins
Distribution of Parameters
• Learn p(b) from training set • If x multivariate gaussian, then
– b gaussian with diagonal covariance
• Can use mixture model for p(b)
)( 1 tb diag λλ L=S ...
CSE586 Robert Collins
Summary so far…
• We can build statistical models of shape change • Require point correspondences across training set • Get compact model (few parameters) by
doing Principle Components Analysis (PCA)
• Next: Matching models to images
CSE586 Robert Collins
Active Shape Models
• Suppose we have a statistical shape model – Trained from sets of examples
• How do we use it to interpret new images? • Use an “Active Shape Model” • Iterative method of matching model to
image (similar to active contours)
CSE586 Robert Collins
Recall : Building Models
• Require labeled training images – landmarks represent correspondences
CSE586 Robert Collins
Recall : Building Shape Models
• Given aligned shapes, { } • Apply PCA
• P – First t eigenvectors of covar. matrix • b – Shape model parameters
ix
Pbxx +=
CSE586 Robert Collins
Recall : Hand Shape Model
1 Varyingb2 Varyingb 3 Varyingb
CSE586 Robert Collins
Active Shape Models
• Match shape model to new image • Require:
– Statistical shape model – Model of image structure at each point
Model Point
Model of Intensity Profile
CSE586 Robert Collins
Placing model in image
• The model points are defined in a model co-ordinate frame
• Must apply global transformation,T, to place in image
Model Frame Image
Pbxx +=
)( PbxX += T),,,;( θsYXT ccx
CSE586 Robert Collins
ASM Search Overview
• Local optimisation • Initialise near target
– Search along profiles for best match,X’ – Update parameters to match to X’.
),( ii YX
CSE586 Robert Collins
Local Structure Models
• Need to search for local match for each point
• Options – Strongest edge – Correlation – Statistical model of profile
CSE586 Robert Collins
Computing Normal to Boundary
Tangent Normal ),(),( xyyx ttnn −=),( yx tt
),( 11 −− ii YX),( 11 ++ ii YX
22
),(),(
yx
yxyx
dd
ddtt
+≈
11
11
−+
−+
−=
−=
iiy
iix
YYdXXd
(Unit vector)
CSE586 Robert Collins
Sampling along profiles
Model boundary
Model point
Profile normal to boundary
Interpolate at these points
,...2,1,0,1,2...
),(),(
−−=
+
insnsiYX ynxn
),( YX
xnns
ynns
ns
),( along length of steps Take yxn nns
CSE586 Robert Collins
Noise reduction
• In noisy images, average orthogonal to profile – Improves signal-to-noise along profile
1ig2ig
3ig
321i 25.05.025.0g Use iii ggg ++=
,...)g,g,g,g,g(...,is profile Sampled
2101-2-=g
CSE586 Robert Collins
Searching for strong edges
)(xg
xdxxdg )(
))1()1((5.0)(−−+= xgxg
dxxdg
Select point along profile at strongest edge
CSE586 Robert Collins
Profile Models
• Sometimes true point not on strongest edge • Model local intensity profile structure to
help locate the point
Strongest edge True position
CSE586 Robert Collins
Statistical Profile Models
• Estimate p.d.f. for sample on profile
• Normalise to allow for global lighting variations
• From training set learn
g
)(gp
CSE586 Robert Collins
Profile Models
• For each point in model – For each training image
• Sample values along profile • Normalise
– Build statistical model • eg Gaussian PDF using eigen-model approach
CSE586 Robert Collins
Searching Along Profiles
• During search we look along a normal for the best match for each profile
)(xg
x
))(( xp g
)(xgForm vector from samples about x
CSE586 Robert Collins
Search algorithm
• Search along profile • Update global transformation, T, and
parameters, b, to minimise
2|)(| PbxX +−T
),( ii YX
CSE586 Robert Collins
Updating parameters
• Find pose and model parameters to minimise
2|),,,;(|),,,,( θθ sYXTsYXf cccc PbxXb +−=
translation rotation
scale shape
CSE586 Robert Collins
Updating Parameters
2|)(|minimise which ),,,( find and Fix PbxXb +−TsYX cc θ
2|)(|minimises which find and ),,,(Fix PbxXb +−TsYX cc θ
))(( 1 xXPb −= −TT
Repeat until convergence:
Analytic solution exists
Iterative Algorithm
Project b onto closest “acceptable” shape
CSE586 Robert Collins
Update step
• Hard constraints
• Soft constraints
• Can also weight by quality of local match
tp)p(T <+− bPbxX subject to |)(| Minimise 2
ii ë||b 3 e.g. ≤
)(log/|)()(| Minimise 2r
21 )p( T bPbxX ++−− σ
CSE586 Robert Collins
Multi-Resolution Search
• Train models at each level of pyramid – Gaussian pyramid with step size 2 – Use same points but different local models
• Start search at coarse resolution – Refine at finer resolution
CSE586 Robert Collins
Gaussian Pyramids
• To generate image at level L – Smooth image at level L-1 with gaussian filter
(eg (1 5 8 5 1)/20) – Sub-sample every other pixel
Each level half the size of the one below
CSE586 Robert Collins
Multi-Resolution Search
• Start at coarse resolution • For each resolution
– Search along profiles for best matches – Update parameters to fit matches – (Apply constraints to parameters) – Until converge at this resolution
CSE586 Robert Collins
Example
Initial pose After 5 iterations Convergence
CSE586 Robert Collins
Active Shape Models
• Advantages – Fast, simple, accurate – Efficient to extend to 3D
• Disadvantages – Only sparse use of image information – Treat local models as independent
CSE586 Robert Collins
State of the Art
Active Appearance Models :
• Triangulated mesh model • Model shape variation using Active Shape Model • also model intensity variation within each patch
Baker and Matthews, http://www.ri.cmu.edu/projects/project_448.html
CSE586 Robert Collins
Active Appearance Models
Baker and Matthews, http://www.ri.cmu.edu/projects/project_448.html
CSE586 Robert Collins
State of the Art
Baker and Matthews, http://www.ri.cmu.edu/projects/project_448.html