1 Applications Shmuel Peleg and Joshua Herman, “Panoramic Mosaics by Manifold Projection”,...

1

Applications

Shmuel Peleg and Joshua Herman, “Panoramic Mosaics by Manifold Projection”, Computer Vision and Pattern Recognition (CVPR), 1997

Wolfgang Heidrich and Hans-Peter Seidel, “View-independent environment maps”, SIGGRAPH / Eurographics Workshop on Graphics Hardware, 1998

Matthew Brand, “Charting a Manifold”, Mitsubishi tech report, 2003

K.Grochow, S. Martin, A. Hertzmann, and Z. PopovicStyle-based Inverse Kinematics, Siggraph 2004

2 Siggraph 2005, 8/1/2005www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html

Applications using manifolds

• Many problems can be phrased in manifold terminology• Provides an alternative way of viewing the

problem• Can also provide some formalism


Applications

•Surface fitting• Consistent parameterization

•Image-based rendering•Environment mapping•Animation


Surface fitting•CT, MRI data, noisy, densely sampled in widely-spaced contours

• Boundary conditions less problematic

Fitting Manifold Surfaces To 3D Point CloudsCindy Grimm, David Laidlaw, and Joseph CriscoJ. of Biomedical Engineering Feb. 2002


Consistent parameterization

•Build manifold once• Fit to multiple bone point data sets


Aligning parameterization

•Identify similar points on meshes• Pin parameterization


Application: Panoramas

•Problem statement:• Given images from a known camera movement

• Rotation about camera axis• “Push-broom” pan (assumes negligible depth)

• “Glue” images together into a single image

Rover, nasa.gov


Camera rotation

•Final image can be rendered on a cylinder• No parallax• Each image samples some number of pixels on

cylinder (manifold) image

Peleg and Herman


Push broom/vertical slit camera•Translation of camera

• Image slit perpendicular to camera motion• Need not travel in straight line

• Depth differences negligible• Parallax

• Manifold is part of ground plane viewed by camera

Ground plane

Direction of travel

Peleg and Herman


Practical problem

•How to line up individual images to create one seamless image?

• Manifold: Final image (3D function RGB on 2D manifold)

• Charts: Individual images (2D charts)• Overlap regions/transition functions: Unknown

• Assume translation• (Account for optical effects of camera)

•Note: Only works for these two camera motions


General solution

•Define a format for the transition function• E.g., translation in x,y

•Define an error metric that measures how well two overlap regions agree

• E.g., pixel difference•Optimize over free parameters in transition function

• E.g., x,y shift between all pairs of overlapping images


Solving for overlaps, transitions

•Find translation that minimizes pixel differences

• Find that minimizes || I0(s) – I1((s))||

• (s) = s + s, where s is unknown

0 1

0 1

s


Final image

•Transition functions align images (abstract manifold)• Final image colors? (RGB function on manifold)

•Blend and embedding functions for each chart• Embedding function: Original image• Blending function: How much to use of each

overlapping image• Usually favor very short blend regions


Application: Environment mapping

•Place scene/model inside sphere•Light intensity/color found by intersecting normal with sphere

• 1-1 mapping between normal direction and sphere• Every point on sphere assigned light intensity/color

•Implementation• Store colors in one (or more) texture maps (2D)


Parameterization

•Surface normal (point on sphere) to point in texture map• Atlas/local parameterization

•Desirable properties• Even sampling of sphere

• Adaptive• Partition

• Overlap (mip mapping, continuity)• Simple to compute

• Amenable to GPU implementation


Approach 1

•Single texture map• Not unique (poles)• Poor sampling• Simple to compute


Approach II

•Cube mapping• Six charts• Discontinuities at edges• Sampling better at center of faces than edges• Simple (plane) computation

• Which plane?


Approach III

•Parabolic mapping• Chart functions use parabolic

function• Better sampling• Slightly more computation• Less-noticeable seams

Heidrich and Hans-Peter Seidel


Proposal

•Use chart approach• Allows for adaptive sampling (more detail where

needed)• Chart sizes uniform: Tile texture map• Include overlap

• Minimal extra texture map• Mip-mapping/down sampling


Application: Animation

•Human configuration space lies on a manifold of dimension n embedded in m dimensional space, where n << m•Articulated skeleton: over 40 degrees of freedom (shoulders, knees, hips, etc., each 1-3 degrees of rotation)•Individual motions (reaching, walking) certainly lie on lower dimension manifolds

• End-point of reach plus time•Shape of manifold of all possible human motions?

• Who knows? K.Grochow, S. Martin, A. Hertzmann, and Z. Popovic


Overview

•Manifold learning• Data samples (e.g., motion capture, key frames)

• Interpolation equals manifold construction• Editing equals manifold editing

• High-level review of manifold learning techniques• What kind of manifolds can you learn/construct

from data?•2D animation example•Manifolds in animation


2D illustration

•Two joint angles• Circle X Circle manifold (torus)

•Animation• Repetitive motion• Joint angle plot

• Circle manifold•Animation is a 1D manifold embedded in 2D


Manifold learning

•Input: Sample points in Rm

• E.g., Motion capture sequence, each pose is a data point, m is number of dof of joints• 2D example: , for each pose

•Assume data lies on a manifold of dimension n• Constraints on manifold shape/geometry (e.g.,

linear, no self-intersections)•Goal: Parameterize manifold

• Single chart


Manifold learning, linear techniques

•Principal components analysis (PCA), Independent components analysis (ICA)

• Assumes shape of manifold in space is linear/planar

• Learn vector(s) that span the hyper-plane


Manifold learning, deformed linear

•Support vector machines (SVM)• Deform space points are in first

• Deformation is “nice”• Warps points to new places• “Guess” which deformation will work

• Learn linear/planar function

Deformation Linear parameterization


Manifold learning, local shape

•Isomap, Local linear embedding (LLE), Semi-definite embedding (SDE)

• Define a distance metric between points• Assumptions:

• K closest neighbors are also closest K neighbors on the manifold

• Disk around point in manifold would contain all K neighbors, but not others

• Know dimension n of manifold• Possible to deform K neighbors into n dimensions while maintaining

(relative) distances

•Output: Single chart/parameterization of entire data set


Isomap, LLE, SDE cont.

•Non-obvious failure modes• Circular/repetitive data sets• Self-intersections

Raw result: 2D embedding

Modified : 1D embedding


Manifold construction as learning

•Use K neighbors to define chart domains (Uc)

• Charts are “squished” Gaussians• Center, tangent vectors

•Find transition functions (affine transformations)• Transformation takes tangent vectors into Rn

• Aligns free vectors with neighbors

UcMatthew Brand, “Charting a Manifold”, Mitsubishi tech report, 2003


Uses of animation manifold•General idea:

• Construct (implicitly or explicitly) a manifold representing valid human poses

• Create a new animation sequence• Foot must touch here, reach here, etc.

• Not sufficient to constrain all degrees of freedom (dof)

• Project on to manifold to fill in remaining dof

K. Grochow, S. L. Martin, A. Hertzmann, Z. Popovic,Style-Based Inverse Kinematics, Siggraph 2004


Re-sequencing as embedded manifold

•Goal: Given existing sequence (samples), add more/change samples•Assumptions:

• Samples come from some smooth manifold• Some form of interpolation gives new samples on

manifold•Current approaches: Interpolation between neighboring samples in sequence for given new time


Re-phrasing problem

•Manifold learning or sequence timing provides parameterization/abstract manifold•Embed manifold with smooth function

• Parameterization• Use function fitting

•Re-sequencing: Evaluate embedding function

E(M)


Caveats

•What makes animation data difficult?• “Distance” loses meaning in >> 10 dimensions

• Every point equally far away• Can’t enumerate

• Noise• Error in capture process• Skeleton only approximates human motion

• Joint angle representation• Don’t explicitly deal with manifold,

parameterization


Summary

•Manifolds provide a formalism for breaking a problem into manageable pieces

• Charts provide local parameterization• Planar domains

• Overlaps: Natural mechanism for moving between parameterizations• Blend functions instead of geometric constraints• No boundary condition problems

•Explicitly encapsulating/representing manifold is beneficial

• Cleaner algorithm specifications

1 Applications Shmuel Peleg and Joshua Herman, “Panoramic Mosaics by Manifold Projection”,...

Documents

Transcript of 1 Applications Shmuel Peleg and Joshua Herman, “Panoramic Mosaics by Manifold Projection”,...