Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… ·...

51
CSE586 Robert Collins CSE 586, Spring 2015 Shape Analysis Statistical Shape Models

Transcript of Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… ·...

Page 1: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

CSE 586, Spring 2015

Shape Analysis Statistical Shape Models

Page 2: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Recall : Building Shape Models

•  Require labeled training images –  landmarks represent correspondences

Page 3: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Recall: Generalized Procrustes Analysis

Page 4: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Examlpe: Hand shape model

•  72 points placed around boundary of hand –  18 hand outlines obtained by thresholding images of

hand on a white background •  Primary landmarks chosen at tips of fingers and

joint between fingers –  Other points placed equally between

1 2 3

4

5 6

Page 5: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Labeling Examples

From Stegmann and Gomez

Page 6: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Labeling Examples

From Stegmann and Gomez

Page 7: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Aligned Training Examples

Page 8: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Point Alignment Training points before and after alignment

Note that each point looks like it varies in position by a Gaussian distribution.

Page 9: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Simple Shape Analysis Fit a Gaussian (mean and covariance) to the cluster of aligned points for each landmark separately.

Problem: Each point DOES NOT move independently from all the other points. In fact, the movement of points is often highly correlated when a shape undergoes deformation.

Page 10: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Statistical Shape Analysis

Goal: Model how the entire collection of points varies. We will concatenate all points from each training example into a single vector in a high dimensional space. The full hand shape model is then built by computing PCA on all model points collectively, which captures correlations of motion between points.

Page 11: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Motivation

Samples from training set, after alignment

Their is information stored in the correlation matrix about the joint correlation of all point movements.

Covariance matrix of all points together as a multivariate Gaussian

Covariance ellipses of each point independently

Page 12: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Page 13: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Page 14: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Page 15: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Page 16: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Page 17: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Page 18: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Page 19: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Page 20: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Page 21: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Hand Shape Model

Varying b1

Varying b2

Varying b3

First 3 modes of variation

Page 22: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Face Shape Example

Page 23: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Distribution of Parameters

•  Learn p(b) from training set •  If x multivariate gaussian, then

– b gaussian with diagonal covariance

•  Can use mixture model for p(b)

)( 1 tb diag λλ L=S ...

Page 24: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Summary so far…

•  We can build statistical models of shape change •  Require point correspondences across training set •  Get compact model (few parameters) by

doing Principle Components Analysis (PCA)

•  Next: Matching models to images

Page 25: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Active Shape Models

•  Suppose we have a statistical shape model – Trained from sets of examples

•  How do we use it to interpret new images? •  Use an “Active Shape Model” •  Iterative method of matching model to

image (similar to active contours)

Page 26: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Active Shape Models

•  Match shape model to new image •  Require:

– Statistical shape model – Model of image structure at each point

Model Point

Model of Intensity Profile

Page 27: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Placing model in image

•  The model points are defined in a model co-ordinate frame

•  Must apply global transformation, T, to place it in the image

Model Frame Image

Pbxx +=

)( PbxX += T),,,;( θsYXT ccx

Page 28: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

ASM Search Overview

•  Local optimisation •  Initialise near target

– Search along profiles for best match X’ – Update parameters to move towards X’.

(X’i , Y’i)

Page 29: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Local Structure Models

•  Need to search for local match for each point

•  Options –  Strongest edge –  Correlation –  Statistical model of profile

Page 30: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Computing Normal to Boundary

Tangent Normal ),(),( xyyx ttnn −=),( yx tt

),( 11 −− ii YX),( 11 ++ ii YX

22

),(),(

yx

yxyx

dd

ddtt

+≈

11

11

−+

−+

−=

−=

iiy

iix

YYdXXd

(Unit vector)

Page 31: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Sampling along profiles

Model boundary

Model point

Profile normal to boundary

Interpolate at these points

,...2,1,0,1,2...

),(),(

−−=

+

i

nsnsiYX ynxn

),( YX

xnns

ynns

ns

),( along length of steps Take yxn nns

Page 32: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Noise reduction

•  In noisy images, average orthogonal to profile –  Improves signal-to-noise along profile

1ig2ig

3ig

321i 25.05.025.0g Use iii ggg ++=

,...)g,g,g,g,g(...,is profile Sampled

2101-2-=g

Page 33: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Searching for strong edges

)(xg

xdxxdg )(

))1()1((5.0)(−−+= xgxg

dxxdg

Select point along profile at strongest edge

Page 34: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Profile Models

•  Sometimes true point not on strongest edge •  Model local intensity profile structure to

help locate the point

Strongest edge True position

Page 35: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Statistical Profile Models

•  Estimate p.d.f. for sample on profile

•  Normalise to allow for global lighting variations

•  From training set learn

g

)(gp

Page 36: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Profile Models

•  For each point in model – For each training image

•  Sample values along profile •  Normalise

– Build statistical model •  eg Gaussian PDF using eigen-model approach

Page 37: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Searching Along Profiles

•  During search we look along a normal for the best match for each profile

)(xg

x

))(( xp g

)(xgForm vector from samples about x

Page 38: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Search algorithm

•  Search along profile •  Update global transformation, T, and

parameters, b, to minimise

2|)(| PbxX +−T

(X’i , Y’i)

Recall from last lecture that we can solve this in closed form!

Page 39: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Updating parameters

•  Find pose and model parameters to minimise

2|),,,;(|),,,,( θθ sYXTsYXf cccc PbxXb +−=

translation rotation

scale shape

Page 40: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Updating Parameters

2|)(|minimise which ),,,( find and Fix PbxXb +−TsYX cc θ

2|)(|minimises which find and ),,,(Fix PbxXb +−TsYX cc θ

Repeat until convergence:

Analytic solution exists

Iterative Algorithm

Problem: solving for b directlyby least squares does not necessarily lead to an “acceptable” shape.

Page 41: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Update step for b

•  Hard constraints

•  Soft constraints

•  Can also weight by quality of local match

tp)p(T <+− bPbxX subject to |)(| Minimise 2

ii ë||b 3 e.g. ≤

)(log/|)()(| Minimise 2r

21 )p( T bPbxX ++−− σ

Page 42: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Update Step for b

•  Intuition: project b onto closest “acceptable” shape.

mean shape

boundary of acceptable shape deviation (e.g. plus or minus 3sigma contour)

unconstrained estimate of shape b

Page 43: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Update Step for b

•  Intuition: project b onto closest “acceptable” shape.

mean shape

boundary of acceptable shape deviation (e.g. plus or minus 3sigma contour)

unconstrained estimate of shape b

two “acceptable” alternatives.

Page 44: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Multi-Resolution Search

•  Train models at each level of pyramid – Gaussian pyramid with step size 2 – Use same points but different local models

•  Start search at coarse resolution – Refine at finer resolution

Page 45: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Gaussian Pyramids

•  To generate image at level L – Smooth image at level L-1 with gaussian filter

(eg (1 5 8 5 1)/20) – Sub-sample every other pixel

Each level half the size of the one below

Page 46: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins Implementation Detail Multi-Resolution Search

•  Start at coarse resolution •  For each resolution

– Search along profiles for best matches – Update parameters to fit matches –  (Apply constraints to parameters) – Until converge at this resolution

Page 47: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Example

Initial pose After 5 iterations Convergence

Page 48: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Active Shape Models

•  Advantages – Fast, simple, accurate – Efficient to extend to 3D

•  Disadvantages – Only sparse use of image information – Treat local models as independent

Page 49: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Active Appearance Models Combined shape and appearance modeling:

• Triangulated mesh model •  Model shape variation using Active Shape Model •  also model intensity variation within each patch

Baker and Matthews, http://www.ri.cmu.edu/projects/project_448.html

Page 50: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Active Appearance Models

Baker and Matthews, http://www.ri.cmu.edu/projects/project_448.html

Page 51: Shape Analysis Statistical Shape Modelsrtc12/CSE586/lectures/cse586ActiveShapeModel… · Statistical Shape Analysis Goal: Model how the entire collection of points varies. We will

CSE586

Robert Collins

Active Appearance Models

Baker and Matthews, http://www.ri.cmu.edu/projects/project_448.html