1

Combining Approximate GeometryCombining Approximate Geometrywith VDTM – A Hybrid Approach towith VDTM – A Hybrid Approach to

3D Video Teleconferencing3D Video Teleconferencing

Celso KurashimaCelso Kurashima22, Ruigang Yang, Ruigang Yang11, Anselmo , Anselmo LastraLastra11

11Department of Computer ScienceDepartment of Computer ScienceUniversity of North Carolina at Chapel HillUniversity of North Carolina at Chapel Hill

22Laboratório de Sistemas Integráveis - LSILaboratório de Sistemas Integráveis - LSIEscola Politécnica da Universidade de São PauloEscola Politécnica da Universidade de São Paulo

Fortaleza, October 8th, 2002Fortaleza, October 8th, 2002

Combining Approximate GeometryCombining Approximate Geometrywith VDTM – A Hybrid Approach towith VDTM – A Hybrid Approach to

3D Video Teleconferencing3D Video Teleconferencing

Celso KurashimaCelso Kurashima22, Ruigang Yang, Ruigang Yang11, Anselmo , Anselmo LastraLastra11

11Department of Computer ScienceDepartment of Computer ScienceUniversity of North Carolina at Chapel HillUniversity of North Carolina at Chapel Hill

22Laboratório de Sistemas Integráveis - LSILaboratório de Sistemas Integráveis - LSIEscola Politécnica da Universidade de São PauloEscola Politécnica da Universidade de São Paulo

Fortaleza, October 8th, 2002Fortaleza, October 8th, 2002

SIBGRAPI 2002

2

IntroductionIntroduction

3

Introduction Introduction

Video ConferenceVideo Conference

2D vs. 3D2D vs. 3D

Video ConferenceVideo Conference

2D vs. 3D2D vs. 3D

1 camera1 camera Many camerasMany cameras

Fixed viewpontFixed viewpont Free viewpointFree viewpoint

No eye contactNo eye contact Eye contactEye contact

Standard VideoStandard Video Computer Vision & Computer Vision & Computer GraphicsComputer Graphics

4

OutlineOutline

• 3D Video Teleconference System3D Video Teleconference System

• Geometry ExtractionGeometry Extraction

• Rendering Images with VDTMRendering Images with VDTM

• ResultsResults

• ConclusionConclusion

• 3D Video Teleconference System3D Video Teleconference System

• Geometry ExtractionGeometry Extraction

• Rendering Images with VDTMRendering Images with VDTM

• ResultsResults

• ConclusionConclusion

5

3D Video Teleconference 3D Video Teleconference System System (1/3)(1/3)

• Top ViewTop View• Top ViewTop View

(A)(A) (B)(B) (A)(A) (B)(B)

PCCLUSTER

PCCLUSTER

PCCLUSTER

PCCLUSTER

6

3D-Video Teleconference System 3D-Video Teleconference System (2/3)(2/3)

Projector

.

.

FirewireCameras

Geometry/Renderer PC

Screen

.

VideoCapture

PC's

LAN

7

3D-Video Teleconference System 3D-Video Teleconference System (3/3)(3/3)

• How does it work?How does it work?

• 11stst: Create a geometry proxy of the : Create a geometry proxy of the personperson

• 22ndnd: Texture map images onto the proxy : Texture map images onto the proxy with VDTM : an IBR method developed with VDTM : an IBR method developed by Debevec by Debevec et alet al..(’96, ’98)(’96, ’98)

• How does it work?How does it work?

• 11stst: Create a geometry proxy of the : Create a geometry proxy of the personperson

• 22ndnd: Texture map images onto the proxy : Texture map images onto the proxy with VDTM : an IBR method developed with VDTM : an IBR method developed by Debevec by Debevec et alet al..(’96, ’98)(’96, ’98)

8

Geometry Proxy Geometry Proxy Extraction Extraction (1/5)(1/5)

• The proxy : a simple geometric The proxy : a simple geometric representation of the personrepresentation of the person

• Mesh of trianglesMesh of triangles

• Map textures onto the triangles facesMap textures onto the triangles faces

• Extraction: a pair of camerasExtraction: a pair of cameras

• May be two of texture camerasMay be two of texture cameras

• The proxy : a simple geometric The proxy : a simple geometric representation of the personrepresentation of the person

• Mesh of trianglesMesh of triangles

• Map textures onto the triangles facesMap textures onto the triangles faces

• Extraction: a pair of camerasExtraction: a pair of cameras

• May be two of texture camerasMay be two of texture cameras

9

Geometry Proxy Extraction Geometry Proxy Extraction (2/5)(2/5)

• Algorithm- Plane + Parallax (Kumar, 94) Algorithm- Plane + Parallax (Kumar, 94) method:method:

– Robust Plane FittingRobust Plane Fitting

– Stereo Feature trackingStereo Feature tracking

• Algorithm- Plane + Parallax (Kumar, 94) Algorithm- Plane + Parallax (Kumar, 94) method:method:

– Robust Plane FittingRobust Plane Fitting

– Stereo Feature trackingStereo Feature tracking

10

Geometry Proxy Extraction Geometry Proxy Extraction (3/5)(3/5)

• Robust Plane FittingRobust Plane Fitting– SegmentationSegmentation

– Points on Points on

the Silhouettethe Silhouette

– MatchingMatching

– Fit PlaneFit Plane

– DistanceDistance

– Std. Dev.Std. Dev.

– Remove distant pointsRemove distant points

– RepeatRepeat

• Robust Plane FittingRobust Plane Fitting– SegmentationSegmentation

– Points on Points on

the Silhouettethe Silhouette

– MatchingMatching

– Fit PlaneFit Plane

– DistanceDistance

– Std. Dev.Std. Dev.

– Remove distant pointsRemove distant points

– RepeatRepeat

LEFT CAMERA RIGHT CAMERALEFT CAMERA RIGHT CAMERA

11

Geometry Proxy Geometry Proxy Extraction Extraction (4/5)(4/5)

Stereo Feature TrackingStereo Feature TrackingKLT tracker (Kanade-Lucas-Tomasi, ‘91, ‘94)KLT tracker (Kanade-Lucas-Tomasi, ‘91, ‘94)

Time

Frame t

S1: trackedfeatures

Camera 0

Camera 1

Frame t

S0: trackedfeatures

Frame t

S1: trackedfeatures

Frame t

S1: trackedfeatures

Camera 0

Camera 1

Frame t

S0: trackedfeatures

Frame t

S0: trackedfeatures

FeatureTracking

Frame t+1

S’0: updatedfeatures

Frame t+1

S’1: updatedfeatures

FeatureTracking

Frame t+1

S’0: updatedfeatures

Frame t+1

S’0: updatedfeatures

Frame t+1

S’1: updatedfeatures

Frame t+1

S’1: updatedfeatures

EpipolarConstraint(EC)

Frame t+1

S’’0: featurespassed EC

Frame t+1

S’’1: featurespassed EC

EpipolarConstraint(EC)

Frame t+1

S’’0: featurespassed EC

Frame t+1

S’’0: featurespassed EC

Frame t+1

S’’1: featurespassed EC

Frame t+1

S’’1: featurespassed EC

Time

Frame t

S1: trackedfeatures

Camera 0

Camera 1

Frame t

S0: trackedfeatures

Frame t

S1: trackedfeatures

Frame t

S1: trackedfeatures

Camera 0

Camera 1

Frame t

S0: trackedfeatures

Frame t

S0: trackedfeatures

FeatureTracking

Frame t+1

S’0: updatedfeatures

Frame t+1

S’1: updatedfeatures

FeatureTracking

Frame t+1

S’0: updatedfeatures

Frame t+1

S’0: updatedfeatures

Frame t+1

S’1: updatedfeatures

Frame t+1

S’1: updatedfeatures

EpipolarConstraint(EC)

Frame t+1

S’’0: featurespassed EC

Frame t+1

S’’1: featurespassed EC

EpipolarConstraint(EC)

Frame t+1

S’’0: featurespassed EC

Frame t+1

S’’0: featurespassed EC

Frame t+1

S’’1: featurespassed EC

Frame t+1

S’’1: featurespassed EC

12

Geometry Proxy Extraction Geometry Proxy Extraction (5/5)(5/5)

Triangulation & Proxy Triangulation & Proxy formation formation

+ =

13

Rendering Images with VDTM – Rendering Images with VDTM – View Dependent Texture View Dependent Texture Mapping Mapping (1/2)(1/2)

• VDTM requires a good spatial geometry VDTM requires a good spatial geometry of the objects of the objects (Debevec98)(Debevec98)

• Our system [ Our system [ Buehler Buehler et alet al (2001) and Heigl (2001) and Heigl et alet al

(1999)(1999) ] ]

Spatial geometry == proxySpatial geometry == proxy

Textures == live images (from Textures == live images (from cameras)cameras)

• Textures are mapped onto the proxy Textures are mapped onto the proxy according to the viewpointaccording to the viewpoint

• VDTM requires a good spatial geometry VDTM requires a good spatial geometry of the objects of the objects (Debevec98)(Debevec98)

• Our system [ Our system [ Buehler Buehler et alet al (2001) and Heigl (2001) and Heigl et alet al

(1999)(1999) ] ]

Spatial geometry == proxySpatial geometry == proxy

Textures == live images (from Textures == live images (from cameras)cameras)

• Textures are mapped onto the proxy Textures are mapped onto the proxy according to the viewpointaccording to the viewpoint

14

Rendering Images with VDTM – Rendering Images with VDTM – View Dependent Texture View Dependent Texture Mapping Mapping (2/2)(2/2)

• Virtual Camera at Virtual Camera at Viewpoint DViewpoint D

• Texture from Texture from cameras Ccameras Cii mapped mapped onto the triangles onto the triangles facesfaces

• Blending weights in Blending weights in vertex Vvertex V

• Angles Angles ii, used to , used to compute the weights compute the weights valuesvalues

ii = exp(- = exp(-ii22/2./2.22))

• Virtual Camera at Virtual Camera at Viewpoint DViewpoint D

• Texture from Texture from cameras Ccameras Cii mapped mapped onto the triangles onto the triangles facesfaces

• Blending weights in Blending weights in vertex Vvertex V

• Angles Angles ii, used to , used to compute the weights compute the weights valuesvalues

ii = exp(- = exp(-ii22/2./2.22))

Subject's 3D proxy points

V

C0 C2 C3C1

0

1

D

2

3

Subject's 3D proxy points

V

C0 C2 C3C1

0

1

D

2

3

15

Results Results (1/3)(1/3)

• Geometry Proxy & Image rendering with VDTMGeometry Proxy & Image rendering with VDTM• Geometry Proxy & Image rendering with VDTMGeometry Proxy & Image rendering with VDTM

• Geometry Geometry ProxyProxy

• Geometry Geometry ProxyProxy

• Image Image rendered with rendered with 2 texture 2 texture camerascameras

• Image Image rendered with rendered with 2 texture 2 texture camerascameras

• Image Image rendered with rendered with 4 texture 4 texture camerascameras

• Image Image rendered with rendered with 4 texture 4 texture camerascameras

16

Results Results (2/3)(2/3)

• Cameras: Firewire IEEE 1394 SONYCameras: Firewire IEEE 1394 SONY– Frame size: 320x240 pixelsFrame size: 320x240 pixels

• Geometry/Renderer PCGeometry/Renderer PC– Processor: Intel Pentium4, 2.2 GHzProcessor: Intel Pentium4, 2.2 GHz

– Graphics card: nVidia GeForce3Graphics card: nVidia GeForce3

• Video Frame Rate: 3-4 fpsVideo Frame Rate: 3-4 fps

• Cameras: Firewire IEEE 1394 SONYCameras: Firewire IEEE 1394 SONY– Frame size: 320x240 pixelsFrame size: 320x240 pixels

• Geometry/Renderer PCGeometry/Renderer PC– Processor: Intel Pentium4, 2.2 GHzProcessor: Intel Pentium4, 2.2 GHz

– Graphics card: nVidia GeForce3Graphics card: nVidia GeForce3

• Video Frame Rate: 3-4 fpsVideo Frame Rate: 3-4 fps

17

Results Results (3/3)(3/3)

• 3D Video Teleconferencing3D Video Teleconferencing

MovieMovie

• 3D Video Teleconferencing3D Video Teleconferencing

MovieMovie

18

ConclusionsConclusions

• A hybrid system for 3D Video A hybrid system for 3D Video TeleconferencingTeleconferencing

• Fast geometry proxy extraction, using a Fast geometry proxy extraction, using a robust plane fitting method and stereo robust plane fitting method and stereo feature tracking, combined with view-feature tracking, combined with view-dependent texture mappingdependent texture mapping

• Real-time demonstration with personal Real-time demonstration with personal computers and commodity graphics cardcomputers and commodity graphics card

• A hybrid system for 3D Video A hybrid system for 3D Video TeleconferencingTeleconferencing

• Fast geometry proxy extraction, using a Fast geometry proxy extraction, using a robust plane fitting method and stereo robust plane fitting method and stereo feature tracking, combined with view-feature tracking, combined with view-dependent texture mappingdependent texture mapping

• Real-time demonstration with personal Real-time demonstration with personal computers and commodity graphics cardcomputers and commodity graphics card

19

Future WorkFuture Work

• Vision-based head-tracking for Vision-based head-tracking for viewpoint controlviewpoint control

• Increase of the frame rate with faster Increase of the frame rate with faster segmentationsegmentation

• Vision-based head-tracking for Vision-based head-tracking for viewpoint controlviewpoint control

• Increase of the frame rate with faster Increase of the frame rate with faster segmentationsegmentation

20

AcknowledgementsAcknowledgements

• Herman TowlesHerman Towles

• Office of the FutureOffice of the Future (OOTF) group at (OOTF) group at UNC-CHUNC-CH

• U.S. Dept. of Energy and Sandia U.S. Dept. of Energy and Sandia National Labs.National Labs.

• U.S. National Science FoundationU.S. National Science Foundation

• NEC/CPDIANEC/CPDIA

• Herman TowlesHerman Towles

• Office of the FutureOffice of the Future (OOTF) group at (OOTF) group at UNC-CHUNC-CH

• U.S. Dept. of Energy and Sandia U.S. Dept. of Energy and Sandia National Labs.National Labs.

• U.S. National Science FoundationU.S. National Science Foundation

• NEC/CPDIANEC/CPDIA

21

Thank you!Thank you!

http://www.cs.unc.edu/Research/ootf/http://www.cs.unc.edu/Research/ootf/

http://www.lsi.usp.br/~kurash/http://www.lsi.usp.br/~kurash/

http://www.cs.unc.edu/Research/ootf/http://www.cs.unc.edu/Research/ootf/

http://www.lsi.usp.br/~kurash/http://www.lsi.usp.br/~kurash/