ComputerVision

Structure from motion

Marc PollefeysCOMP 256

Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …

ComputerVision Last time: Optical Flow

0 tyx IvIuI uIx

It

Ixu

Ixu=It

Aperture problem:

two solutions:- regularize (smoothness prior)- constant over window (i.e. Lucas-Kanade)

Coarse-to-fine, parametric models, etc…

ComputerVision

Aug 26/28 - Introduction

Sep 2/4 Cameras Radiometry

Sep 9/11 Sources & Shadows Color

Sep 16/18 Linear filters & edges

(Isabel hurricane)

Sep 23/25 Pyramids & Texture Multi-View Geometry

Sep30/Oct2 Stereo Project proposals

Oct 7/9 Tracking (Welch) Optical flow

Oct 14/16 - -

Oct 21/23 Silhouettes/carving (Fall break)

Oct 28/30 - Structure from motion

Nov 4/6 Project update Camera calibration

Nov 11/13 Segmentation Fitting

Nov 18/20 Prob. segm.&fit. Matching templates

Nov 25/27 Matching relations (Thanksgiving)

Dec 2/4 Range data Final project

Tentative class schedule

ComputerVision Today’s menu

• Affine structure from motion– Geometric construction– Factorization

• Projective structure from motion– Factorization– Sequential

ComputerVision

Affine Structure from Motion

Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990). 1990 Optical Society of America.

Given m pictures of n points, can we recover• the three-dimensional configuration of these points?• the camera configurations?

(structure)(motion)

ComputerVision

Orthographic Projection

Parallel Projection

ComputerVision Weak-Perspective Projection

Paraperspective Projection

ComputerVision The Affine Structure-from-Motion Problem

Given m images of n fixed points P we can write

Problem: estimate the m 2x4 matrices M andthe n positions P from the mn correspondences p .

i

j ij

2mn equations in 8m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

j

ComputerVision

The Affine Ambiguity of Affine SFM

If M and P are solutions, i j

So are M’ and P’ wherei j

and

Q is an affinetransformation.

When the intrinsic and extrinsic parameters are unknown

ComputerVision

Affine Spaces: (Semi-Formal) Definition

ComputerVision Example: R as an Affine Space

2

ComputerVision In General

The notation

is justified by the fact that choosing some origin O in Xallows us to identify the point P with the vector OP.

Warning: P+u and Q-P are defined independently of O!!

ComputerVision Barycentric Combinations

• Can we add points? R=P+Q NO!

• But, when we can define

• Note:

ComputerVision Affine Subspaces

ComputerVision

Affine Coordinates

• Coordinate system for U:

• Coordinate system for Y=O+U:

• Coordinate system for Y:

• Affine coordinates:

• Barycentric coordinates:

ComputerVision When do m+1 points define a p-dimensional subspace Y

of an n-dimensional affine space X equipped with some coordinate frame basis?

Writing that all minors of size (p+2)x(p+2) of D are equal to zero gives the equations of Y.

Rank ( D ) = p+1, where

ComputerVision Affine Transformations

Bijections from X to Y that:• map m-dimensional subspaces of X onto m-dimensional subspaces of Y;• map parallel subspaces onto parallel subspaces; and• preserve affine (or barycentric) coordinates.

In E they are combinations of rigid transformations, non-uniform scalings and shears.

Bijections from X to Y that:• map lines of X onto lines of Y; and• preserve the ratios of signed lengths of line segments.

3

ComputerVision Affine Transformations II

• Given two affine spaces X and Y of dimension m, and two coordinate frames (A) and (B) for these spaces, there exists a unique affine transformation mapping (A) onto (B).

• Given an affine transformation from X to Y, one can always write:

• When coordinate frames have been chosen for X and Y,this translates into:

ComputerVision Affine projections induce affine transformations from planes

onto their images.

ComputerVision Affine Shape

Two point sets S and S’ in some affine space X are affinely equivalent when there exists an affine transformation : X X such that X’ = ( X ).

Affine structure from motion = affine shape recovery.

= recovery of the corresponding motion equivalence classes.

ComputerVision

Geometric affine scene reconstruction from two images(Koenderink and Van Doorn, 1991).

ComputerVision Affine Structure from Motion

(Koenderink and Van Doorn, 1991)

Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990). 1990 Optical Society of America.

ComputerVision

The Affine Epipolar Constraint

Note: the epipolar lines are parallel.

ComputerVision Affine Epipolar Geometry

ComputerVision The Affine Fundamental Matrix

where

ComputerVision

An Affine Trick.. Algebraic Scene Reconstruction

ComputerVision The Affine Structure of Affine Images

Suppose we observe a scene with m fixed cameras..

The set of all images of a fixed scene is a 3D affine space!

ComputerVision

has rank 4!

ComputerVision From Affine to Vectorial Structure

Idea: pick one of the points (or their center of mass)as the origin.

ComputerVision

What if we could factorize D? (Tomasi and Kanade, 1992)

Affine SFM is solved!

Singular Value Decomposition

We can take

ComputerVision From uncalibrated to calibrated cameras

Weak-perspective camera:

Calibrated camera:

Problem: what is Q ?

Note: Absolute scale cannot be recovered. The Euclidean shape(defined up to an arbitrary similitude) is recovered.

ComputerVision

Reconstruction Results (Tomasi and Kanade, 1992)

Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi andT. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.

ComputerVision More examples

Tomasi Kanade’92,Poelman & Kanade’94

ComputerVision More examples

Tomasi Kanade’92,Poelman & Kanade’94

ComputerVision More examples

Tomasi Kanade’92,Poelman & Kanade’94

ComputerVision Further Factorization work

Factorization with uncertainty

Factorization for indep. moving objects

Factorization for dynamic objects

Perspective factorization (next week)

Factorization with outliers and missing pts.

(Irani & Anandan, IJCV’02)

(Costeira and Kanade ‘94)

(Bregler et al. 2000, Brand 2001)

(Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective))

(Sturm & Triggs 1996, …)

ComputerVision Multiple indep. moving objects

ComputerVision Multiple indep. moving objects

ComputerVision Dynamic structure from motion

Extend factorization approaches to deal with dynamic shapes

(Bregler et al ’00; Brand ‘01)

ComputerVision Representing dynamic shapes

represent dynamic shape as varying linear combination of basis shapes

k

kk (t)ScS(t)

(fig. M.Brand)

ComputerVision Projecting dynamic shapes

PtScR k

kk

(figs. M.Brand)Rewrite:

ComputerVision Dynamic image sequences

One image:

Multiple images(figs. M.Brand)

ComputerVision Dynamic SfM factorization?

Problem: find J so that M has proper structure

ComputerVision Dynamic SfM factorization

(Bregler et al ’00)

Assumption: SVD preserves order and orientation of basis shape components

ComputerVision Results

(Bregler et al ’00)

ComputerVision Dynamic SfM factorization

(Brand ’01)

constraints to be satisfied for M

constraints to be satisfied for M, use to compute J

hard!

(different methods are possible, not so simple and also not optimal)

ComputerVision Non-rigid 3D subspace flow

• Same is also possible using optical flow in stead of features, also takes uncertainty into account

(Brand ’01)

ComputerVision Results

(Brand ’01)

ComputerVision

(Brand ’01)

Results

ComputerVision Results

(Bregler et al ’01)

ComputerVision Next class:

Projective structure from motion