Space Time Tracking ECCV 2002
description
Transcript of Space Time Tracking ECCV 2002
![Page 1: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/1.jpg)
Space Time Tracking Space Time Tracking ECCV 2002ECCV 2002
Lorenzo TorresaniLorenzo Torresani
Christoph BreglerChristoph Bregler
![Page 2: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/2.jpg)
OutlineOutline
ProblemProblemBackgroundBackgroundStructure from MotionStructure from MotionMatrix DecompositionMatrix DecompositionNon-Rigid Motion EstimationNon-Rigid Motion EstimationNon-Rigid Shapes EstimationNon-Rigid Shapes EstimationResultsResults
![Page 3: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/3.jpg)
ProblemProblem
““To track feature points on non-rigid objects To track feature points on non-rigid objects without using any prior model”without using any prior model”
![Page 4: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/4.jpg)
Rank of a MatrixRank of a Matrix
N
M
Rank (A) = Number of linearly independent vectors in
N
M
Rank (B) = Number of linearly independent vectors in
For M x N the Rank of A ≤ min (M,N)
![Page 5: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/5.jpg)
Rank of a Matrix Rank of a Matrix cont’dcont’d
Rank C = ?
Columns of C are Linear combination of Columns of A
There are only N independent vectors in A Rank C = N
![Page 6: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/6.jpg)
SVDSVD
SVD for a matrix A writes A as a product of three matrices:
• U • D• V
• Every m x n matrix has a singular value decomposition
Am x n
Um x n
Dn x n
VT
n x n
U,V have orthogonalcolumns
![Page 7: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/7.jpg)
Frame 1 Frame 2 ………… Frame F
Tomasi Kanade Structure from MotionTomasi Kanade Structure from Motion
Given N 2D trajectories taken over F images, recover 3D Given N 2D trajectories taken over F images, recover 3D structure and motion (Camera pose)structure and motion (Camera pose)
• Assumption:• 3D Object is rigid• Orthographic Projection
• Tracks can be computed using any standard tracker (KLT etc)
![Page 8: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/8.jpg)
Tomasi Kanade Structure from Motion cont’dTomasi Kanade Structure from Motion cont’d
Assume a set of P 3D points on a rigid object (structure)
S = [P1, P2 …….. PP ]
Orthographic Projection
where (u,v) are image coordinates and M is orthographic projection where (u,v) are image coordinates and M is orthographic projection matrixmatrix
Subtract mean of all u’s and v’s to center the world coordinate frame at the Subtract mean of all u’s and v’s to center the world coordinate frame at the center of the object.center of the object. This will get rid of T in the above equationThis will get rid of T in the above equation
![Page 9: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/9.jpg)
2D coordinates of N points over F images can be 2D coordinates of N points over F images can be written in one matrixwritten in one matrix
W is called the measurement/tracking matrixW is called the measurement/tracking matrix
Rank 3
Tomasi Kanade Structure from Motion cont’dTomasi Kanade Structure from Motion cont’d
![Page 10: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/10.jpg)
From W to R and SFrom W to R and S
Force the rank of W to be 3
SVD
![Page 11: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/11.jpg)
![Page 12: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/12.jpg)
StepsSteps
Matrix Decomposition of W matrix for non-Matrix Decomposition of W matrix for non-rigid objectsrigid objects
Estimate Motion Matrix using reliable set Estimate Motion Matrix using reliable set of pointsof points
Estimate shape basis (S) for all other Estimate shape basis (S) for all other feature points (unreliable)feature points (unreliable)
![Page 13: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/13.jpg)
For Non Rigid ConstraintFor Non Rigid Constraint
![Page 14: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/14.jpg)
3D Non Rigid Shape Model 3D Non Rigid Shape Model
Linear Combination of K Basis ShapesLinear Combination of K Basis Shapes Each basis shape is SEach basis shape is Sii 3 x P3 x P matrix describing P points matrix describing P points
S1 S2………………
SK
= l1S1 l2S2+ + … + lKSKS
Courtesy Christopher Bregler
![Page 15: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/15.jpg)
Matrix DecompositionMatrix Decomposition Project P points of shape SProject P points of shape S Scaled Orthographic ProjectionScaled Orthographic Projection
Move world coordinate to object centeroid (This will get Move world coordinate to object centeroid (This will get rid of T)rid of T)
W Q M
![Page 16: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/16.jpg)
-
![Page 17: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/17.jpg)
Tracking MatrixTracking Matrix
Complete 2D Tracks or Flow
Rank of W 3K
2F x 3K 3K x P
In Tomasi Kanade it was 3
![Page 18: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/18.jpg)
Non Rigid Motion EstimationNon Rigid Motion Estimation
Since Since W W isis rank-deficient, rank-deficient, Q Q can be estimated w/o can be estimated w/o the full availability of the full availability of WW
r <= 3Kr <= 3K point tracks will span the space of the point tracks will span the space of the trajectories of all the points (as rank of W is r)trajectories of all the points (as rank of W is r)
? = ?
W Q’ M’
known reliable tracks
Courtesy Christopher Bregler
![Page 19: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/19.jpg)
r = 9
![Page 20: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/20.jpg)
Trajectory ConstraintTrajectory Constraint
Q’
3D positions of point i for the K modes of deformation
t=2
t=1
t=F
. . . .= .
...frames
miwi : full trajectory
• Generate m trajectories (hypothesis) using Factored Sampling• Evaluate w by computing sum of square difference around point i.
Courtesy Christopher Bregler
![Page 21: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/21.jpg)
• Each column mi of unreliable M is computed as expected value of posterior.
![Page 22: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/22.jpg)
![Page 23: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/23.jpg)
ResultsResults
![Page 24: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/24.jpg)
![Page 25: Space Time Tracking ECCV 2002](https://reader036.fdocuments.net/reader036/viewer/2022081520/568152d9550346895dc0f55c/html5/thumbnails/25.jpg)