MASKS © 2004 Invitation to 3D vision

Lecture 8Segmentation of Dynamical

Scenes

MASKS © 2004 Invitation to 3D vision

One-body two-views

MASKS © 2004 Invitation to 3D vision

One-body multiple-views

MASKS © 2004 Invitation to 3D vision

Motivation and problem statement

•A static scene: multiple 2D motion models

•A dynamic scene: multiple 3D motion models

•Given an image sequence, determine Number of motion models (affine, Euclidean, etc.) Motion model: affine (2D) or Euclidean (3D) Segmentation: model to which each pixel belongs

MASKS © 2004 Invitation to 3D vision

Previous work on 2D motion segmentation

• Local methods (Wang-Adelson ’93) Estimate one model per

pixel using data in a window Cluster models with K-

means Iterate Aperture problem Motion across boundaries

• Global methods (Irani-Peleg ‘92) Dominant motion: fit one

motion model to all pixels Look for misaligned pixels

& fit a new model to them Iterate

• Normalized cuts (Shi-Malik ‘98) Similarity matrix

based on motion profile

Segment pixels using eigenvector

MASKS © 2004 Invitation to 3D vision

Previous work on 3D motion segmentation

• Factorization techniques Orthographic/discrete: Costeira-Kanade ’98, Gear ‘98 Perspective/continuous: Vidal-Soatto-Sastry ’02 Omnidirectional/continuous: Shakernia-Vidal-Sastry ’03

• Special cases: Points in a line (orth-discrete): Han and Kanade ’00 Points in a conic (perspective): Avidan-Shashua ’01 Points in a line (persp.-continuous): Levin-Shashua ’01

• 2-body case: Wolf-Shashua ‘01

MASKS © 2004 Invitation to 3D vision

Previous work: probabilistic techniques

• Probabilistic approaches Generative model: data membership + motion model Obtain motion models using Expectation Maximization

E-step: Given motion models, segment image data M-step: Given data segmentation, estimate motion models

• 2D Motion Segmentation Layered representation (Jepson-Black’93, Ayer-Sawhney ’95,

Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski-Anandan ’99)

• 3D Motion Segmentation EM+Reprojection Error: Feng-Perona’98 EM+Model Selection: Torr ’98

• How to initialize iterative algorithms?

MASKS © 2004 Invitation to 3D vision

Our approach to motion segmentation

•This work considers full perspective projection multiple objects general motions

•We show that Problem is equivalent to

polynomial factorization There is a unique global

closed form solution if n<5

Exact solution is obtained using linear algebra

Can be used to initialize EM-based algorithms

Image points

Number of motions

Multibody Fund. Matrix

Epipolar lines

Epipoles

Fundamental Matrices

Motion segmentation

Multi epipolar lines

Multi epipole

MASKS © 2004 Invitation to 3D vision

One-dimensional segmentation

•Number of models?

MASKS © 2004 Invitation to 3D vision

One-dimensional segmentation

• For n groups

• Number of groups

• Groups

MASKS © 2004 Invitation to 3D vision

One-dimensional segmentation

• Solution is unique if

• Solution is closed form if and only if

• Solves the eigenvector segmentation problem e.g. normalized cuts

MASKS © 2004 Invitation to 3D vision

Three-dimensional motion segmentation

Generalized PCA (Vidal, Ma ’02)

• Solve for the roots of a polynomial of degree in one variable

• Solve for a linear system in variables

MASKS © 2004 Invitation to 3D vision

The multibody epipolar constraint

• Rotation:• Translation:• Epipolar constraint

• Multiple motions

Multibody epipolar constraint• Satisfied by ALL points regardless of segmentation

• Segmentation is algebraically eliminated!!!

MASKS © 2004 Invitation to 3D vision

The multibody fundamental matrix

Lifting EmbeddingEmbedding

Bilinear on embedded data!

• Veronese map (polynomial embedding)

• Multibody fundamental matrix

MASKS © 2004 Invitation to 3D vision

Estimation of the number of motions

• Theorem: Given image points corresponding to motions, if at least 8 points correspond to each object, then

1 2 43

8 35 99 225Minimum number of

points

MASKS © 2004 Invitation to 3D vision

Estimation of multibody fundamental matrix

1-body motion n-body motion

MASKS © 2004 Invitation to 3D vision

Segmentation of fundamental matrices

Given

rank condition for n motions

linear system F

Multibody epipolar transfer

Multibody epipole

Fundamental matrices

MASKS © 2004 Invitation to 3D vision

Multibody epipolar transfer

Multibody epipolar line

Lifting

Polynomial factorization

MASKS © 2004 Invitation to 3D vision

Multibody epipole

• The multibody epipole is the solution of the linear system

• Epipoles are obtained using polynomial factorization

Lifting

Number of distinct epipoles

MASKS © 2004 Invitation to 3D vision

From images to epipoles

MASKS © 2004 Invitation to 3D vision

Fundamental matrices

• Columns of are epipolar lines

• Polynomial factorization to compute them up to scale

• Scales can be computed linearly

MASKS © 2004 Invitation to 3D vision

The multibody 8-point algorithm

Image point

Multibody epipolar line

Multibody epipolar transfer

Polynomial Factorization

Linear system

Polynomial Factorization

Epipolar lines

Multibody epipole

Epipoles

Embedded image point

Veronese map

Fundamental matrix

Linear system

MASKS © 2004 Invitation to 3D vision

Optimal 3D motion segmentation

• Zero-mean Gaussian noise• Constrained optimization problem on

• Optimal function for 1 motion

• Optimal function for n motions

• Solved using Riemanian Gradien Descent

MASKS © 2004 Invitation to 3D vision

Comparison of 1 body and n bodies

MASKS © 2004 Invitation to 3D vision

Other cases: linearly moving objects

• Multibody epipole• Recovery of epipoles• Fundamental matrices• Feature segmentation 1 2 105

2 5 20 65

1 2 43

8 35 99 225

Minimum number of

points

MASKS © 2004 Invitation to 3D vision

Other cases: affine flows

Affine motion segmentation:

constant brightness constraint

3D motion segmentation:epipolar constraint

• In linear motions, geometric constraints are linear

• Two-view motion constraints could be bilinear!!!

MASKS © 2004 Invitation to 3D vision

3D motion segmentation results

N = 44 + 48 + 81 = 173

MASKS © 2004 Invitation to 3D vision

Results

MASKS © 2004 Invitation to 3D vision

Results

MASKS © 2004 Invitation to 3D vision

More results

MASKS © 2004 Invitation to 3D vision

Conclusions

• There is an analytic solution to 3D motion segmentation based on Multibody epipolar constraint: it does not depend

on the segmentation of the data Polynomial factorization: linear algebra Solution is closed form iff n<5

• A similar technique also applies to Eigenvector segmentation: from similarity

matrices Generalized PCA: mixtures of subspaces 2-D motion segmentation: of affine motions

• Future work Reduce data complexity, sensitivity analysis,

robustness

MASKS © 2004 Invitation to 3D vision

References

• R. Vidal, Y. Ma, S. Soatto and S. Sastry. Two-view multibody structure from motion, International Journal of Computer Vision, 2004

• R. Vidal and S. Sastry. Optimal segmentation of dynamic scenes from two perspective views, International Conference on Computer Vision and Pattern Recognition, 2003

• R. Vidal and S. Sastry. Segmentation of dynamic scenes from image intensities, IEEE Workshop on Vision and Motion Computing, 2002.