Convergence of vision and graphics

Convergence of vision and graphics

Jitendra Malik

University of California at Berkeley

Jitendra Malik

University of California at Berkeley

Overview Overview

3D capture:Modeling,simulation

Rendering Display

Applications:Simulation

Virtual RealityRemote collaboration

Graphics and VisionGraphics and Vision

• Computer graphics is the forward problem: given scene geometry, reflectances and lighting, synthesize an image.

• Computer vision must address the inverse problem: given an image/multiple images, reconstruct the scene geometry, reflectacnes and illumination.

• Computer graphics is the forward problem: given scene geometry, reflectances and lighting, synthesize an image.

• Computer vision must address the inverse problem: given an image/multiple images, reconstruct the scene geometry, reflectacnes and illumination.

Image-based Modeling Image-based Modeling

– Vary viewpoint– Vary lighting– Vary scene configuration

– Vary viewpoint– Vary lighting– Vary scene configuration

Recover Models of Real World Scenes and Make Possible Various Visual Interactions

Image-based Modeling Image-based Modeling • 1st Generation---- vary viewpoint but not

lighting– Acquire photographs

– Recover geometry (explicit or implicit)

– Texture map

• 1st Generation---- vary viewpoint but not lighting– Acquire photographs

– Recover geometry (explicit or implicit)

– Texture map

Recovering geometryRecovering geometry

• Historical roots in photogrammetry and analysis of 3D cues in human vision

• Single images adequate given knowledge of object class

• Multiple images make the problem easier, but not trivial as corresponding points must be identified.

• Historical roots in photogrammetry and analysis of 3D cues in human vision

• Single images adequate given knowledge of object class

• Multiple images make the problem easier, but not trivial as corresponding points must be identified.

Arc de Triomphe

Arc de Triomphe

The Taj Mahal

Taj MahalTaj Mahalmodeled frommodeled from

one photographone photographby G. Borshukovby G. Borshukov

Campus Model of UC Berkeley Campus Model of UC Berkeley

Campanile + 40 Buildings (Debevec et al)Campanile + 40 Buildings (Debevec et al)

Image-based ModelingImage-based Modeling• 2nd Generation---- vary viewpoint and

lighting– Recover geometry & reflectance properties

– Render using light transport simulation or local shading

• 2nd Generation---- vary viewpoint and lighting– Recover geometry & reflectance properties

– Render using light transport simulation or local shading

Original Lighting & Viewpoint Novel Lighting & Viewpoint

Inverse Global Illumination (Yu et al)Inverse Global Illumination (Yu et al)

Reflectance Properties

Radiance Maps

Geometry Light Sources

Real vs. Synthetic Real vs. Synthetic

Real vs. Synthetic Real vs. Synthetic

Image-based ModelingImage-based Modeling

• 3rd Generation--Vary spatial configurations in addition to viewpoint and lighting

• 3rd Generation--Vary spatial configurations in addition to viewpoint and lighting

Novel Viewpoint Novel Viewpoint & Configuration

Our FrameworkOur Framework

• Input– Multiple range scans of

a scene

– Multiple photographs of the same scene

• Input– Multiple range scans of

a scene

– Multiple photographs of the same scene

• Output– Geometric meshes of

each object in the scene

– Registered texture maps for objects

• Output– Geometric meshes of

each object in the scene

– Registered texture maps for objects

Segmentation: From images to objects

Segmentation: From images to objects

Segmentation Results[Yu, Ferencz and Malik ’00]

Segmentation Results[Yu, Ferencz and Malik ’00]

Models of Individual ObjectsModels of Individual Objects

Texture-Mapping and Object Manipulation

Texture-Mapping and Object Manipulation

Image Based modeling for motion capture

Body Suits, Markers Video Motion Capture

Eadweard Muybridge [Bregler and Malik ’98]

Continuing Challenges Continuing Challenges

• Finding correspondences automatically

• Optimal estimation of structure from n views under perspective projection

• Models of reflectance and texture for natural materials and objects

• Finding correspondences automatically

• Optimal estimation of structure from n views under perspective projection

• Models of reflectance and texture for natural materials and objects