John A. Bender

Applications in Real-time 3D Tracking

Collaborators:Pietro Perona

Luis GoncalvesKen Goldberg

Karl ChenIlan Lobel

Steve Nowlin

Inspiration

Application

(image: Lorenzo Wang Graphics)

Tracking Problem

Camera Design

Orange Micro iBOT IEEE 1394(FireWire) webcam - $100 custom modifications

Camera Design

Orange Micro iBOT IEEE 1394(FireWire) webcam - $100 custom modifications

• 3-port FireWire PCI card has total bandwidth of 400 Mb/s - $100

• 320x240 pixels, 3 color channels, 8 bits/pixel, 30 fps, 3 cameras= 166 Mb/s

• isochronous sampling??

• processing time??

Image Processing

Camera Modeling

real-world image

lens

projectedimage

Camera Modeling

real-world image

lens

projectedimage

refractionboundary

Camera Modeling

focal lengthprincipal

point

parameters of interest…

camera matrix:

Fx 0 Px

0 Fy Py

0 0 1

point on image plane: [ (x-Px)/Fx (y-Py)/Fy 1 ] (homogeneous coordinate)

Camera Calibration

3D Modeling

transformations between camerasare modeled with rotation/translationcomponents

translation vector: [ Tx Ty Tz ]rotation vector: [ Rx Ry Rz ]

rotation matrix:X0X1 X0Y1 X0Z1

Y0X1 Y0Y1 Y0Z1

Z0X1 Z0Y1 Z0Z1

3D Modeling

transform rays into a common coordinate system and intersect

transformation for a homogeneous 3D point (z=1): X0 = R10*X1 + T10

Rig Calibration

Image Processing

• Microsoft DirectX (DirectShow) interfaces with hardware

• image callback executed on every frame with timestamp,each camera runs in a separate thread

• background modeling / foreground masking withIntel Performance Primitives optimized image processing libraries

• ellipse fitting using Intel Open Computer Vision libraries

• triangulation including ellipse size

• fish determination

• TCP network streaming

• executes at ~20 Hz on a Dell 2.8 GHz Pentium 4 with1.5 GB of 533 MHz DDR SDRAM

Path Determination

each point is a 3D point whichcould contain a fish

each column is a different time point

Path Determination

draw paths from each point infirst frame to each point in second frame

Path Determination

draw paths from each point infirst frame to each point in second frame

extend each path to all possible positions

Path Determination

extend each path to all possible positions

cull paths with too-low traversal probability

Path Determination

add dummy node with fixed transitionprobability in case of disappearances

Path Determination

add dummy node with fixed transitionprobability in case of disappearances

place dummy node at last known position

Path Determination

for each ending node, choose the uniquepath that minimizes traversal probability

Path Determination

Rendering

• virtual fish tank

• each virtual fish follows the position of a real fish

• Newtonian motion model for virtual fish ensures smooth movement

• camera position in virtual tank set to head of gold fish

Results – from Vision…Results – from Vision to Reality