ComputerVision
Structure from motion
Marc PollefeysCOMP 256
Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …
ComputerVision Last time: Optical Flow
0 tyx IvIuI uIx
It
Ixu
Ixu=It
Aperture problem:
two solutions:- regularize (smoothness prior)- constant over window (i.e. Lucas-Kanade)
Coarse-to-fine, parametric models, etc…
ComputerVision
Aug 26/28 - Introduction
Sep 2/4 Cameras Radiometry
Sep 9/11 Sources & Shadows Color
Sep 16/18 Linear filters & edges
(Isabel hurricane)
Sep 23/25 Pyramids & Texture Multi-View Geometry
Sep30/Oct2 Stereo Project proposals
Oct 7/9 Tracking (Welch) Optical flow
Oct 14/16 - -
Oct 21/23 Silhouettes/carving (Fall break)
Oct 28/30 - Structure from motion
Nov 4/6 Project update Camera calibration
Nov 11/13 Segmentation Fitting
Nov 18/20 Prob. segm.&fit. Matching templates
Nov 25/27 Matching relations (Thanksgiving)
Dec 2/4 Range data Final project
Tentative class schedule
ComputerVision Today’s menu
• Affine structure from motion– Geometric construction– Factorization
• Projective structure from motion– Factorization– Sequential
ComputerVision
Affine Structure from Motion
Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990). 1990 Optical Society of America.
Given m pictures of n points, can we recover• the three-dimensional configuration of these points?• the camera configurations?
(structure)(motion)
ComputerVision The Affine Structure-from-Motion Problem
Given m images of n fixed points P we can write
Problem: estimate the m 2x4 matrices M andthe n positions P from the mn correspondences p .
i
j ij
2mn equations in 8m+3n unknowns
Overconstrained problem, that can be solvedusing (non-linear) least squares!
j
ComputerVision
The Affine Ambiguity of Affine SFM
If M and P are solutions, i j
So are M’ and P’ wherei j
and
Q is an affinetransformation.
When the intrinsic and extrinsic parameters are unknown
ComputerVision In General
The notation
is justified by the fact that choosing some origin O in Xallows us to identify the point P with the vector OP.
Warning: P+u and Q-P are defined independently of O!!
ComputerVision Barycentric Combinations
• Can we add points? R=P+Q NO!
• But, when we can define
• Note:
ComputerVision
Affine Coordinates
• Coordinate system for U:
• Coordinate system for Y=O+U:
• Coordinate system for Y:
• Affine coordinates:
• Barycentric coordinates:
ComputerVision When do m+1 points define a p-dimensional subspace Y
of an n-dimensional affine space X equipped with some coordinate frame basis?
Writing that all minors of size (p+2)x(p+2) of D are equal to zero gives the equations of Y.
Rank ( D ) = p+1, where
ComputerVision Affine Transformations
Bijections from X to Y that:• map m-dimensional subspaces of X onto m-dimensional subspaces of Y;• map parallel subspaces onto parallel subspaces; and• preserve affine (or barycentric) coordinates.
In E they are combinations of rigid transformations, non-uniform scalings and shears.
Bijections from X to Y that:• map lines of X onto lines of Y; and• preserve the ratios of signed lengths of line segments.
3
ComputerVision Affine Transformations II
• Given two affine spaces X and Y of dimension m, and two coordinate frames (A) and (B) for these spaces, there exists a unique affine transformation mapping (A) onto (B).
• Given an affine transformation from X to Y, one can always write:
• When coordinate frames have been chosen for X and Y,this translates into:
ComputerVision Affine Shape
Two point sets S and S’ in some affine space X are affinely equivalent when there exists an affine transformation : X X such that X’ = ( X ).
Affine structure from motion = affine shape recovery.
= recovery of the corresponding motion equivalence classes.
ComputerVision
Geometric affine scene reconstruction from two images(Koenderink and Van Doorn, 1991).
ComputerVision Affine Structure from Motion
(Koenderink and Van Doorn, 1991)
Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990). 1990 Optical Society of America.
ComputerVision The Affine Structure of Affine Images
Suppose we observe a scene with m fixed cameras..
The set of all images of a fixed scene is a 3D affine space!
ComputerVision From Affine to Vectorial Structure
Idea: pick one of the points (or their center of mass)as the origin.
ComputerVision
What if we could factorize D? (Tomasi and Kanade, 1992)
Affine SFM is solved!
Singular Value Decomposition
We can take
ComputerVision From uncalibrated to calibrated cameras
Weak-perspective camera:
Calibrated camera:
Problem: what is Q ?
Note: Absolute scale cannot be recovered. The Euclidean shape(defined up to an arbitrary similitude) is recovered.
ComputerVision
Reconstruction Results (Tomasi and Kanade, 1992)
Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi andT. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.
ComputerVision Further Factorization work
Factorization with uncertainty
Factorization for indep. moving objects
Factorization for dynamic objects
Perspective factorization (next week)
Factorization with outliers and missing pts.
(Irani & Anandan, IJCV’02)
(Costeira and Kanade ‘94)
(Bregler et al. 2000, Brand 2001)
(Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective))
(Sturm & Triggs 1996, …)
ComputerVision Dynamic structure from motion
Extend factorization approaches to deal with dynamic shapes
(Bregler et al ’00; Brand ‘01)
ComputerVision Representing dynamic shapes
represent dynamic shape as varying linear combination of basis shapes
k
kk (t)ScS(t)
(fig. M.Brand)
ComputerVision Dynamic SfM factorization
(Bregler et al ’00)
Assumption: SVD preserves order and orientation of basis shape components
ComputerVision Dynamic SfM factorization
(Brand ’01)
constraints to be satisfied for M
constraints to be satisfied for M, use to compute J
hard!
(different methods are possible, not so simple and also not optimal)
ComputerVision Non-rigid 3D subspace flow
• Same is also possible using optical flow in stead of features, also takes uncertainty into account
(Brand ’01)
Top Related