CSCE 641 Computer Graphics: Image-based Modeling

CSCE 641 Computer Graphics: Image-based Modeling

Jinxiang Chai

Outline

Stereo matching - Traditional stereo

- Multi-baseline stereo

- Active stereo

Volumetric stereo - Visual hull

- Voxel coloring

- Space carving

Multi-baseline stereo reconstruction

Need to choose reference frames!

Does not fully utilize pixel information across all images

Volumetric stereo

Scene VolumeScene Volume

VV

Input ImagesInput Images

(Calibrated)(Calibrated)

Goal: Goal: Determine occupancy, “color” of points in VDetermine occupancy, “color” of points in V

Discrete formulation: Voxel Coloring

Discretized Discretized


Input ImagesInput Images

(Calibrated)(Calibrated)

Goal: Assign RGBA values to voxels in Vphoto-consistent with images

Complexity and computability

Discretized Discretized


N voxelsN voxels

C colorsC colors

33

All Scenes (CN3)Photo-Consistent

Scenes

TrueScene

Issues

Theoretical Questions• Identify class of all photo-consistent scenes

Practical Questions• How do we compute photo-consistent models?

1. C=2 (shape from silhouettes)• Volume intersection [Baumgart 1974]

> For more info: Rapid octree construction from image sequences. R. Szeliski, CVGIP: Image Understanding, 58(1):23-32, July 1993. (this paper is apparently not available online) or

> W. Matusik, C. Buehler, R. Raskar, L. McMillan, and S. J. Gortler, Image-Based Visual Hulls, SIGGRAPH 2000 ( pdf 1.6 MB )

2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]

3. General Case• Space carving [Kutulakos & Seitz 98]

Voxel coloring solutions

http://graphics.lcs.mit.edu/~wojciech/vh/IBVH2000.pdf

Why use silhouettes?

Can be computed robustly

Can be computed efficiently

- =

background background + +

foregroundforeground

backgroundbackground foreground foreground

Reconstruction from silhouettes (C = 2)

Binary ImagesBinary Images

How to construct 3D volume?

Reconstruction from silhouettes (C = 2)

Binary ImagesBinary Images

Approach: • Backproject each silhouette

• Intersect backprojected volumes

Volume intersection

Reconstruction Contains the True Scene• But is generally not the same

• In the limit (all views) get visual hull > Complement of all lines that don’t intersect S

Voxel algorithm for volume intersection

Color voxel black if on silhouette in every image• for M images, N3 voxels

• Don’t have to search 2N3 possible scenes!

O( ? ),

Visual Hull Results

Download data and results from

http://www-cvr.ai.uiuc.edu/ponce_grp/data/visual_hull/



Properties of volume intersection

Pros• Easy to implement, fast

• Accelerated via octrees [Szeliski 1993] or interval techniques [Matusik 2000]

Cons• No concavities

• Reconstruction is not photo-consistent

• Requires identification of silhouettes


1. C=2 (silhouettes)• Volume intersection [Baumgart 1974]


> For more info: http://www.cs.washington.edu/homes/seitz/papers/ijcv99.pdf


http://www.cs.washington.edu/homes/seitz/papers/ijcv99.pdf

1. Choose voxel1. Choose voxel2. Project and correlate2. Project and correlate

3.3. Color if consistentColor if consistent(standard deviation of pixel

colors below threshold)

Voxel coloring approach

Visibility Problem: Visibility Problem: in which images is each voxel visible?in which images is each voxel visible?

The visibility problem

Inverse Visibility?known images

Unknown SceneUnknown Scene

Which points are visible in which images?

Known SceneKnown Scene

Forward Visibility - Z-buffer

- Painter’s algorithm

known scene

LayersLayers

Depth ordering: visit occluders first!

SceneScene

TraversalTraversal

Condition: Condition: depth order is the depth order is the same for all input viewssame for all input views

Panoramic depth ordering

• Cameras oriented in many different directions

• Planar depth ordering does not apply

Panoramic depth ordering

Layers radiate outwards from camerasLayers radiate outwards from cameras

Panoramic layering

Layers radiate outwards from camerasLayers radiate outwards from cameras

Compatible camera configurations

Depth-Order Constraint• Scene outside convex hull of camera centers

Outward-Lookingcameras inside scene

Inward-Lookingcameras above scene

Calibrated image acquisition

Calibrated Turntable360° rotation (21 images)

Selected Dinosaur ImagesSelected Dinosaur Images

Selected Flower ImagesSelected Flower Images

Voxel coloring results

Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI

Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI

Limitations of depth ordering

A view-independent depth order may not exist

p q

Need more powerful general-case algorithms• Unconstrained camera positions

• Unconstrained scene geometry/topology


1. C=2 (silhouettes)• Volume intersection [Baumgart 1974]



> For more info: http://www.cs.washington.edu/homes/seitz/papers/kutu-ijcv00.pdf

http://www.cs.washington.edu/homes/seitz/papers/kutu-ijcv00.pdf

Space carving algorithm

Space Carving Algorithm

Image 1 Image N

…...

• Initialize to a volume V containing the true scene

• Repeat until convergence

• Choose a voxel on the current surface

• Carve if not photo-consistent

• Project to visible input images

Which shape do you get?

The Photo Hull is the UNION of all photo-consistent scenes in V• It is a photo-consistent scene reconstruction

• Tightest possible bound on the true scene

True SceneTrue Scene

VV

Photo HullPhoto Hull

VV

Space carving algorithm

The Basic Algorithm is Unwieldy• Complex update procedure

Alternative: Multi-Pass Plane Sweep• Efficient, can use texture-mapping hardware

• Converges quickly in practice

• Easy to implement

Results Algorithm

Multi-pass plane sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence

True Scene Reconstruction

Multi-pass plane sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence

Space carving results: african violet

Input Image (1 of 45) Reconstruction

ReconstructionReconstruction

Space carving results: hand

Input Image(1 of 100)

Views of Reconstruction

Properties of space carving

Pros• Voxel coloring version is easy to implement, fast

• Photo-consistent results

• No smoothness prior

Cons• Bulging

• No smoothness prior

Next lecture

Images user input range

scans

Model

Images

Image based modeling

Image-based renderingGeometry+ Images

Geometry+ Materials

Images + Depth

Light field

Panoroma

Kinematics

Dynamics

Etc.

Camera + geometry

Stereo reconstruction

Given two or more images of the same scene or object, compute a representation of its shape

knownknowncameracamera

viewpointsviewpoints

Stereo reconstruction

Given two or more images of the same scene or object, compute a representation of its shape

knownknowncameracamera

viewpointsviewpoints

How to estimate camera parameters?

- where is the camera?

- where is it pointing?

- what are internal parameters, e.g. focal length?

Spectrum of IBMR

Images user input range

scans

Model

Images

Image based modeling

Image-based renderingGeometry+ Images

Geometry+ Materials

Images + Depth

Light field

Panoroma

Kinematics

Dynamics

Etc.

Camera + geometry

How can we estimate the camera parameters?

Camera calibration

Augmented pin-hole camera - focal point, orientation

- focal length, aspect ratio, center, lens distortion

Known 3DKnown 3D

Classical calibration - 3D 2D

- correspondence

Camera calibration online resources

http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#links

Camera and calibration target

Classical camera calibration

Known 3D coordinates and 2D coordinates - known 3D points on calibration targets

- find corresponding 2D points in image using feature detection

algorithm

Camera parameters

u0

v0

100-sy0

sx аuv1

Perspective proj. View trans.Viewport proj.

Known 3D coords and 2D coordsKnown 3D coords and 2D coords

Camera parameters

u0

v0

100-sy0

sx аuv1

Perspective proj. View trans.Viewport proj.

Known 3D coords and 2D coordsKnown 3D coords and 2D coords

Intrinsic camera parameters (5 parameters)

extrinsic camera parameters (6 parameters)

Camera matrix

Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into acamera matrix

M = K [R | t ]

(put 1 in lower r.h. corner for 11 d.o.f.)

Camera matrix calibration

Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)


Linear regression:• Bring denominator over, solve set of (over-determined) linear

equations. How?


Linear regression:• Bring denominator over, solve set of (over-determined) linear

equations. How?

• Least squares (pseudo-inverse) - 11 unknowns (up to scale) - 2 equations per point (homogeneous coordinates) - 6 points are sufficient

Nonlinear camera calibration

Perspective projection:

1100

0

1 3

2

1

3

2

1

0

0

i

i

i

T

T

T

y

x

i

i

z

y

x

t

t

t

r

r

r

vf

uf

v

u



1100

0

1 3

2

1

3

2

1

0

0

i

i

i

T

T

T

y

x

i

i

z

y

x

t

t

t

r

r

r

vf

uf

v

u

K R T P



2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters:

1100

0

1 3

2

1

3

2

1

0

0

i

i

i

T

T

T

y

x

i

i

z

y

x

t

t

t

r

r

r

vf

uf

v

u

K R T P

33

32302

33

30213021

)(

)(

tPr

ttfPrvrfv

tPr

tuttfPrurrfu

T

yTT

yi

Tx

TTTx

i



2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters:

1100

0

1 3

2

1

3

2

1

0

0

i

i

i

T

T

T

y

x

i

i

z

y

x

t

t

t

r

r

r

vf

uf

v

u

33

32302

33

30213021

)(

)(

tPr

ttfPrvrfv

tPr

tuttfPrurrfu

T

yTT

yi

Tx

TTTx

i

K

);,,( iPTRKf

);,,( iPTRKg

R T P

Multiple calibration images

Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M

for i=1,…,N

This can be formulated as a nonlinear optimization problem:

);,,(

);,,(

ijjji

ijjji

PTRKgv

PTRKfu

M

j

N

iijj

jiijj

ji PTRKgvPTRKfu

1 1

22 ));,,(());,,((

Multiple calibration images

Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N

This can be formulated as a nonlinear optimization problem:

);,,(

);,,(

ijjji

ijjji

PTRKgv

PTRKfu

M

j

N

iijj

jiijj

ji PTRKgvPTRKfu

1 1

22 ));,,(());,,((

Solve the optimization using nonlinear optimization techniques:

- Gauss-newton

- Levenberg-Marquardt

http://en.wikipedia.org/wiki/Levenberg-Marquardt_algorithm



Nonlinear approach

Advantages:• can solve for more than one camera pose at a time

• fewer degrees of freedom than linear approach

• Standard technique in photogrammetry, computer vision, computer graphics

- [Tsai 87] also estimates lens distortions (freeware @ CMU)http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/v-source.html

Disadvantages:• more complex update rules

• need a good initialization (recover K [R | t] from M)

How can we estimate the camera parameters?

Application: camera calibration for sports video

[Farin et. Al]

images Court model

Calibration from 2D motion

Structure from motion (SFM) - track points over a sequence of images

- estimate for 3D positions and camera positions

- calibrate intrinsic camera parameters before hand

Self-calibration: - solve for both intrinsic and extrinsic camera parameters

SFM = Holy Grail of 3D Reconstruction

Take movie of object

Reconstruct 3D model

Would be

commercially

highly viable

How to get feature correspondences

Feature-based approach

- good for images

- feature detection (corners)

- feature matching using RANSAC (epipolar line)

Pixel-based approach

- good for video sequences

- patch based registration with lucas-kanade algorithm

- register features across the entire sequence

Structure from motion

Two Principal Solutions• Bundle adjustment (nonlinear optimization)

• Factorization (SVD, through orthographic approximation, affine geometry)

Nonlinear approach for SFM

What’s the difference between camera calibration and SFM?


M

j

N

iijj

jiijj

ji

TRK

PTRKgvPTRKfujj

1 1

22

}{},{,

));,,(());,,((minarg


- camera calibration: known 3D and 2D


M

j

N

iijj

jiijj

ji

TRKP

PTRKgvPTRKfujji

1 1

22

}{},{,},{

)),,,(()),,,((minarg

M

j

N

iijj

jiijj

ji

TRK

PTRKgvPTRKfujj

1 1

22

}{},{,

));,,(());,,((minarg



- SFM: unknown 3D and known 2D


M

j

N

iijj

jiijj

ji

TRKP

PTRKgvPTRKfujji

1 1

22

}{},{,},{

)),,,(()),,,((minarg

M

j

N

iijj

jiijj

ji

TRK

PTRKgvPTRKfujj

1 1

22

}{},{,

));,,(());,,((minarg



- SFM: unknown 3D and known 2D

- what’s 3D-to-2D registration problem?

SFM: bundle adjustment

SFM = Nonlinear Least Squares problem

Minimize through• Gradient Descent

• Conjugate Gradient

• Gauss-Newton

• Levenberg Marquardt common method

Prone to local minima

M

j

N

iijj

jiijj

ji

TRKP

PTRKgvPTRKfujji

1 1

22

}{},{,},{

)),,,(()),,,((minarg

Count # Constraints vs #Unknowns

M camera poses

N points

2MN point constraints

6M+3N unknowns

Suggests: need 2mn 6m + 3n

But: Can we really recover all parameters???

M

j

N

iijj

jiijj

ji

TRKP

PTRKgvPTRKfujji

1 1

22

}{},{,},{

)),,,(()),,,((minarg

Count # Constraints vs #Unknowns

M camera poses

N points

2MN point constraints

6M+3N unknowns (known intrinsic camera parameters)

Suggests: need 2mn 6m + 3n

But: Can we really recover all parameters???• Can’t recover origin, orientation (6 params)

• Can’t recover scale (1 param)

Thus, we need 2mn 6m + 3n - 7

M

j

N

iijj

jiijj

ji

TRKP

PTRKgvPTRKfujji

1 1

22

}{},{,},{

)),,,(()),,,((minarg

Are we done?

No, bundle adjustment has many local minima.

SFM using Factorization

12

1

2

1

i

i

i

T

T

i

i

z

y

x

t

t

r

r

v

u

Assume an othorgraphic camera

Image World


12

1

2

1

i

i

i

T

T

i

i

z

y

x

t

t

r

r

v

u

Assume othorgraphic camera

Image World

i

i

i

T

T

N

ii

i

N

ii

i

z

y

x

r

r

N

vv

N

uu

2

1

1

1

Subtract the mean


N

N

N

T

T

N

N

z

y

x

z

y

x

z

y

x

r

r

v

u

v

u

v

u

...

...

...

~

~

...

...~

~

~

~

2

2

2

1

1

1

2

1

2

2

1

1

Stack all the features from the same frame:


N

N

N

T

T

N

N

z

y

x

z

y

x

z

y

x

r

r

v

u

v

u

v

u

...

...

...

~

~

...

...~

~

~

~

2

2

2

1

1

1

2

1

2

2

1

1

N

N

N

TF

TF

T

T

NF

NF

F

F

F

F

NF

NF

F

F

F

F

z

y

x

z

y

x

z

y

x

r

r

r

r

v

u

v

u

v

u

v

u

v

u

v

u

...

...

...

~

~

...

...~

~

~

~

~

~

...

...~

~

~

~

2

2

2

1

1

1

2,

1,

2,1

1,1

,

,

2,

2,

1,

1,

,

,

2,

2,

1,

1,


Stack all the features from all the images:

W


N

N

N

T

T

N

N

z

y

x

z

y

x

z

y

x

r

r

v

u

v

u

v

u

...

...

...

~

~

...

...~

~

~

~

2

2

2

1

1

1

2

1

2

2

1

1

N

N

N

TF

TF

T

T

NF

NF

F

F

F

F

NF

NF

F

F

F

F

z

y

x

z

y

x

z

y

x

r

r

r

r

v

u

v

u

v

u

v

u

v

u

v

u

...

...

...

~

~

...

...~

~

~

~

~

~

...

...~

~

~

~

2

2

2

1

1

1

2,

1,

2,1

1,1

,

,

2,

2,

1,

1,

,

,

2,

2,

1,

1,

NFW 2

~



W

32 FM NS 3


N

N

N

TF

TF

T

T

NF

NF

F

F

F

F

NF

NF

F

F

F

F

z

y

x

z

y

x

z

y

x

r

r

r

r

v

u

v

u

v

u

v

u

v

u

v

u

...

...

...

~

~

...

...~

~

~

~

~

~

...

...~

~

~

~

2

2

2

1

1

1

2,

1,

2,1

1,1

,

,

2,

2,

1,

1,

,

,

2,

2,

1,

1,

NFW 2

~32 FM


W

NS 3

Factorize the matrix into two matrix using SVD:

NFW 2

~

TNF

TNF VSUMVUW 2

1

32

1

322

~~~


N

N

N

TF

TF

T

T

NF

NF

F

F

F

F

NF

NF

F

F

F

F

z

y

x

z

y

x

z

y

x

r

r

r

r

v

u

v

u

v

u

v

u

v

u

v

u

...

...

...

~

~

...

...~

~

~

~

~

~

...

...~

~

~

~

2

2

2

1

1

1

2,

1,

2,1

1,1

,

,

2,

2,

1,

1,

,

,

2,

2,

1,

1,

NFW 2

~32 FM


NS 3


NFW 2

~

TNF

TNF VSUMVUW 2

1

32

1

322

~~~

NNFF SQSQMM

31

333333232

~~


N

N

N

TF

TF

T

T

NF

NF

F

F

F

F

NF

NF

F

F

F

F

z

y

x

z

y

x

z

y

x

r

r

r

r

v

u

v

u

v

u

v

u

v

u

v

u

...

...

...

~

~

...

...~

~

~

~

~

~

...

...~

~

~

~

2

2

2

1

1

1

2,

1,

2,1

1,1

,

,

2,

2,

1,

1,

,

,

2,

2,

1,

1,

NFW 2

~32 FM


W

NS 3


NFW 2

~

TNF

TNF VSUMVUW 2

1

32

1

322

~~~

NNFF SQSQMM

31

333333232

~~

How to compute the matrix ? 33Q


2,2,2,11,1

2,

1,

2,1

1,1

3232 FF

TF

TF

T

T

TFF rrrr

r

r

r

r

MM

M is the stack of rotation matrix:

2,2,

2,1,

1,2,

1,1,

2,12,1

2,11,1

1,12,1

1,11,1

FTF

FTF

FTF

FTF

T

T

T

T

rr

rr

rr

rr

rr

rr

rr

rr


2,2,2,11,1

2,

1,

2,1

1,1

3232 FF

TF

TF

T

T

TFF rrrr

r

r

r

r

MM


2,2,

2,1,

1,2,

1,1,

2,12,1

2,11,1

1,12,1

1,11,1

FTF

FTF

FTF

FTF

T

T

T

T

rr

rr

rr

rr

rr

rr

rr

rr

1 010

1 010

Orthogonal constraints from rotation matrix


2,2,2,11,1

2,

1,

2,1

1,1

3232 FF

TF

TF

T

T

TFF rrrr

r

r

r

r

MM

2,2,

2,1,

1,2,

1,1,

2,12,1

2,11,1

1,12,1

1,11,1

FTF

FTF

FTF

FTF

T

T

T

T

rr

rr

rr

rr

rr

rr

rr

rr


1 010

1 010


TF

TF MQQM 32333332

~~


TF

TF MQQM 32333332

~~

2,2,

2,1,

1,2,

1,1,

2,12,1

2,11,1

1,12,1

1,11,1

FTF

FTF

FTF

FTF

T

T

T

T

rr

rr

rr

rr

rr

rr

rr

rr

1 010

1 010

Orthogonal constraints from rotation matrices:


TF

TF MQQM 32333332

~~

2,2,

2,1,

1,2,

1,1,

2,12,1

2,11,1

1,12,1

1,11,1

FTF

FTF

FTF

FTF

T

T

T

T

rr

rr

rr

rr

rr

rr

rr

rr

1 010

1 010


QQ: symmetric 3 by 3 matrix


TF

TF MQQM 32333332

~~

2,2,

2,1,

1,2,

1,1,

2,12,1

2,11,1

1,12,1

1,11,1

FTF

FTF

FTF

FTF

T

T

T

T

rr

rr

rr

rr

rr

rr

rr

rr

1 010

1 010


How to compute QQ?

least square solution

- 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix)



TF

TF MQQM 32333332

~~

2,2,

2,1,

1,2,

1,1,

2,12,1

2,11,1

1,12,1

1,11,1

FTF

FTF

FTF

FTF

T

T

T

T

rr

rr

rr

rr

rr

rr

rr

rr

1 010

1 010


How to compute QQ?

least square solution

- 4F linear constraints, 9 unknowns (6 independent due to symmetric matrix) How to compute Q from QQ:

SVD again: 2

1

UQVUQQ T



2,2,2,11,1

2,

1,

2,1

1,1

3232 FF

TF

TF

T

T

TFF rrrr

r

r

r

r

MM

2,2,

2,1,

1,2,

1,1,

2,12,1

2,11,1

1,12,1

1,11,1

FTF

FTF

FTF

FTF

T

T

T

T

rr

rr

rr

rr

rr

rr

rr

rr


1 010

1 010


TF

TF MQQM 32333332

~~


Computing QQ is easy:

- 3F linear equations

- 6 independent unknowns

SFM using factorization

1. Form the measurement matrix

2. Decompose the matrix into two matrices and using SVD

3. Compute the matrix Q with least square and SVD

4. Compute the rotation matrix and shape matrix:

and

NFW 2

~

NS 3

~ 32

~FM

QMM F 32

~ 32

1 ~

FSQS

Weak-perspective Projection

Factorization also works for weak-perspective projection (scaled orthographic projection):

d z0

12

1

2

1

i

i

i

T

T

i

i

z

y

x

t

t

r

r

v

u

SFM using factorization

Bundle adjustment (nonlinear optimization) - work with perspective camera model - work with incomplete data - prone to local minima

Factorization: - closed-form solution for weak perspective camera - simple and efficient - usually need complete data - becomes complicated for full-perspective camera model

Phil Torr’s structure from motion toolkit in matlab (click here)

Voodoo camera tracker (click here)

http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/TORR1/index.html

http://www.digilab.uni-hannover.de/docs/manual.html

Results from SFM

[Han and Kanade]

All together video

Click here

- feature detection

- feature matching (epipolar geometry)

- structure from motion

- stereo reconstruction

- triangulation

- texture mapping

http://video.google.com/videoplay?docid=4218558799314950359&q=structure+from+motion+video&total=419&start=0&num=10&so=0&type=search&plindex=7

Challenging objects

Why are they so difficult to model?

How to model them?

CSCE 641 Computer Graphics: Image-based Modeling

Documents

Transcript of CSCE 641 Computer Graphics: Image-based Modeling