CSE 586, Spring 2010 Advanced Computer Visionrtc12/CSE586Spring2010/lectures/cse586...CSE 586,...

CSE586 Robert Collins

CSE 586, Spring 2010 Advanced Computer Vision

Procrustes Shape Analysis PART 2: Active Shape Models


Lecture material from Tim Cootes University of Manchester. For more info, see http://www.isbe.man.ac.uk/~bim/ (includes code for exploring active shape/appearance models).

Credits

lots of slides are due to


Generalized Procrustes Analysis


Shape Space

Consider K landmark points (xi,yi) in 2D

Dimension of the space of observations is 2*K

But what is dimension of the space of the shapes?

Recall, two point configurations are equivalent (have the same shape) if one can be translated, rotated and scaled into the other.

Thus, each unique “shape” is an equivalence class.


Shape Space

The shape space is then the quotient space induced by this equivalence class on the observation space

So shape space has dimension 2*K - 4

This is often called Kendall’s Shape Space.

k

rotation translation scaling


Shape Space

For triangles (K=3), Kendall’s shape space can be visualized as a sphere.

For more points, the space is a more complicated manifold (is it a hypersphere?)

Rather than measure variation in shape on the sphere (or higher dimensional nonlinear manifold), it is common to project points into the tangent plane taken at the mean shape. This is a linear space, so can use PCA, Gaussians, mixture of Gaussians, etc.


Shape Space

http://www.york.ac.uk/res/fme/resources/morphologika/helpfiles/shapespace.html


Shape Space

Tangent projection can be carried out by subtracting the vector of normalized shape coordinates of a figure from the mean vector, and taking just the component perpendicular to the mean.

tangent plane

mean shape

nonlinear shape space

shape sample

shape sample projected into tangent plane


Shape Space

If shape variation is relatively small, we can just take the shape difference vector as an approximation to the vector in the tangent plane.

tangent plane

mean shape

nonlinear shape space

shape sample


Aligned Shapes

•  Note: need to model the aligned shapes

x

space shape

We talked about this two lectures ago


Statistical Shape Models

•  For shape synthesis – Parameterised model preferable

•  For image matching we can get away with only knowing p(b) – Usually more efficient to reduce dimensionality

where possible

)(bx shapef= Pbxx += e.g.


Dimensionality Reduction

•  Coordinates often correlated •  Nearby points move together

11bpxx +≈

1b

xx

1p


Principal Component Analysis

•  Compute eigenvectors of covariance,S •  Eigenvectors : main directions •  Eigenvalue : variance along eigenvector

1p2p

1λ∝ 2λ∝We talked about this last time


Dimensionality Reduction

•  Data lies in subspace of reduced dim.

•  However, for some t,

iλ

i

tjbj >≈ if 0

t

) is of (Variance jjb λ

nnbb ppxx +++= L11 ...


Building Shape Models

•  Given aligned shapes, { } •  Apply PCA

•  P – First t eigenvectors of covar. matrix •  b – Shape model parameters

ix

Pbxx +≈


Hand shape model

•  72 points placed around boundary of hand –  18 hand outlines obtained by thresholding images of

hand on a white background

•  Primary landmarks chosen at tips of fingers and joint between fingers –  Other points placed equally between

1 2 3

4

5 6


Labeling Examples

From Stegmann and Gomez


Point Alignment


Training points before and after alignment


Hand Shape Model

Samples from training set, after alignment


Component Analysis


Full hand shape model is built by computing PCA on all model points collectively, which captures correlations of motion between points.

Independent PCA on each data point cluster


Hand Shape Model

Varying b1

Varying b2

Varying b3

First 3 modes of variation


Face Shape Example

Sample face training image

1st mode of variation


Distribution of Parameters

•  Learn p(b) from training set •  If x multivariate gaussian, then

– b gaussian with diagonal covariance

•  Can use mixture model for p(b)

)( 1 tb diag λλ L=S ...


Summary so far…

•  We can build statistical models of shape change •  Require point correspondences across training set •  Get compact model (few parameters) by

doing Principle Components Analysis (PCA)

•  Next: Matching models to images


Active Shape Models

•  Suppose we have a statistical shape model – Trained from sets of examples

•  How do we use it to interpret new images? •  Use an “Active Shape Model” •  Iterative method of matching model to

image (similar to active contours)


Recall : Building Models

•  Require labeled training images –  landmarks represent correspondences


Recall : Building Shape Models

•  Given aligned shapes, { } •  Apply PCA

•  P – First t eigenvectors of covar. matrix •  b – Shape model parameters

ix

Pbxx +=


Recall : Hand Shape Model

1 Varyingb2 Varyingb 3 Varyingb


Active Shape Models

•  Match shape model to new image •  Require:

– Statistical shape model – Model of image structure at each point

Model Point

Model of Intensity Profile


Placing model in image

•  The model points are defined in a model co-ordinate frame

•  Must apply global transformation,T, to place in image

Model Frame Image

Pbxx +=

)( PbxX += T),,,;( θsYXT ccx


ASM Search Overview

•  Local optimisation •  Initialise near target

– Search along profiles for best match,X’ – Update parameters to match to X’.

),( ii YX


Local Structure Models

•  Need to search for local match for each point

•  Options –  Strongest edge –  Correlation –  Statistical model of profile


Computing Normal to Boundary

Tangent Normal ),(),( xyyx ttnn −=),( yx tt

),( 11 −− ii YX),( 11 ++ ii YX

22

),(),(

yx

yxyx

dd

ddtt

+≈

11

11

−+

−+

−=

−=

iiy

iix

YYdXXd

(Unit vector)


Sampling along profiles

Model boundary

Model point

Profile normal to boundary

Interpolate at these points

,...2,1,0,1,2...

),(),(

−−=

+

insnsiYX ynxn

),( YX

xnns

ynns

ns

),( along length of steps Take yxn nns


Noise reduction

•  In noisy images, average orthogonal to profile –  Improves signal-to-noise along profile

1ig2ig

3ig

321i 25.05.025.0g Use iii ggg ++=

,...)g,g,g,g,g(...,is profile Sampled

2101-2-=g


Searching for strong edges

)(xg

xdxxdg )(

))1()1((5.0)(−−+= xgxg

dxxdg

Select point along profile at strongest edge


Profile Models

•  Sometimes true point not on strongest edge •  Model local intensity profile structure to

help locate the point

Strongest edge True position


Statistical Profile Models

•  Estimate p.d.f. for sample on profile

•  Normalise to allow for global lighting variations

•  From training set learn

g

)(gp


Profile Models

•  For each point in model – For each training image

•  Sample values along profile •  Normalise

– Build statistical model •  eg Gaussian PDF using eigen-model approach


Searching Along Profiles

•  During search we look along a normal for the best match for each profile

)(xg

x

))(( xp g

)(xgForm vector from samples about x


Search algorithm

•  Search along profile •  Update global transformation, T, and

parameters, b, to minimise

2|)(| PbxX +−T

),( ii YX


Updating parameters

•  Find pose and model parameters to minimise

2|),,,;(|),,,,( θθ sYXTsYXf cccc PbxXb +−=

translation rotation

scale shape


Updating Parameters

2|)(|minimise which ),,,( find and Fix PbxXb +−TsYX cc θ

2|)(|minimises which find and ),,,(Fix PbxXb +−TsYX cc θ

))(( 1 xXPb −= −TT

Repeat until convergence:

Analytic solution exists

Iterative Algorithm

Project b onto closest “acceptable” shape


Update step

•  Hard constraints

•  Soft constraints

•  Can also weight by quality of local match

tp)p(T <+− bPbxX subject to |)(| Minimise 2

ii ë||b 3 e.g. ≤

)(log/|)()(| Minimise 2r

21 )p( T bPbxX ++−− σ


Multi-Resolution Search

•  Train models at each level of pyramid – Gaussian pyramid with step size 2 – Use same points but different local models

•  Start search at coarse resolution – Refine at finer resolution


Gaussian Pyramids

•  To generate image at level L – Smooth image at level L-1 with gaussian filter

(eg (1 5 8 5 1)/20) – Sub-sample every other pixel

Each level half the size of the one below


Multi-Resolution Search

•  Start at coarse resolution •  For each resolution

– Search along profiles for best matches – Update parameters to fit matches –  (Apply constraints to parameters) – Until converge at this resolution


Example

Initial pose After 5 iterations Convergence


Active Shape Models

•  Advantages – Fast, simple, accurate – Efficient to extend to 3D

•  Disadvantages – Only sparse use of image information – Treat local models as independent


State of the Art

Active Appearance Models :

• Triangulated mesh model •  Model shape variation using Active Shape Model •  also model intensity variation within each patch

Baker and Matthews, http://www.ri.cmu.edu/projects/project_448.html


Active Appearance Models



State of the Art


CSE 586, Spring 2010 Advanced Computer Visionrtc12/CSE586Spring2010/lectures/cse586...CSE 586,...

Documents

Transcript of CSE 586, Spring 2010 Advanced Computer Visionrtc12/CSE586Spring2010/lectures/cse586...CSE 586,...