Cameras and Projectors

CS335 Principles of Multimedia Systems

Cameras and Projectors

Hao Jiang

Computer Science Department

Boston College

Oct. 2, 2007


Cameras and Projectors

Cameras and projectors have been intensively used in many different multimedia applications.

It is important to understand techniques to manipulate these devices.

We will study basic methods about– calibration,– image warping and blending,– and other issues in building a camera/projector system.


Example Application: Projector Geometry Distortion

Compensation

Automatically correcting projector geometry distortion usinga camera and projector system.


Camera / Projector Geometry Model

Camera and projector can be modeled as a pinhole imaging system.

Object PointOptical Center

Image Plane

Image point

Focal length

Opticalaxis


Camera Geometry Model

A camera projects a 3D point onto a 2D point in a image.

(X’,Y’,Z’)

x = (f X’/Z’)/dx + Ox = fx X’/Z’ + Ox

y = (f Y’/Z’)/dy + Oy = fy Y’/Z’ + Oy

in camera’s frame is

dx and dy are width and height of an imagepixel

f

(Ox, Oy)

3D point (X, Y, Z)x

y


Homogenous Coordinate

The homogenous coordinate of a 3D point (x,y,z) is(X,Y,Z,W) where X/W = X, Y/W=Y and Z/W=Z.

The homogenous coordinate of a 2D point (x,y) is (X,Y,W) where X/W = x, Y/W=Y.

We can convert a 3D point from one frame to another, by simply T*P, where T is a 4x4 matrix and P is the homogeneous coordinate of the 3D point.


Camera Matrix

Using homogenous coordinate, the camera projection can be represented as

xyw

~ A3x4

XYZW

Where A3x4 is a 3 rows and 4 columns matrix, called

camera matrix.

p = = A P a11 a12 a13 a14

a21 a22 a23 a24

a31 a32 a33 a34

P =


Projection from a Plane to Another Plane

O

p

P

p = A P

X

Y

Z

Since We have

xyw

~a11 a12 a14

a21 a22 a24

a31 a32 a34

XY1

= p = H p’

H is defines a Homography.


The Projector Model

Projector shares similar model with a camera.

The only difference is that a projector projects a flat object that emits lights based on a computer image onto an image plane, the screen.

Based on the previous analysis about planar object projection, the image from the computer and the one projected on the screen are related by a homography.


The Camera-Projector System

Image sent to projector Camera image

Image on the screenH1*p

H2*r

pq

r

H2*H1*p


Calibration

We would like to compute H1 and H2.

We project marker points on the screen and form equations

a11Xn + a12Yn + a14

a31Xn

+ a32Yn + a34

a21Xn + a22Yn

+ a24

a31Xn

+ a32Yn + a34

= xn

= yn

n = 1 .. N

a34=1


Image Adjustment (I)

Projector image Camera image

Image on the screen

H1*p H2*H1*p

Pre-warpingp q

Each p is projected to the camera image and the color is determinedby color interpolation in the desired image.

Assume thatthe viewer isclose to the camera.

The desiredImage.


Image Adjustment (II)

Project image Camera image

Image on the screen

H1*p H2*H1*p

Pre-warping

Estimate H2 using screen corners projections.

Pre-warp image based on the desired image on the screen.

The desired image


Projection onto Arbitrary Surfaces

We have learned how to deal with projector distortion using a planar screen.

We can extend the method into other types of surfaces, such as cylinder or spheres.

We need a relative dense mesh grid to capture the local deformation model.


Titled Large Screen Display


Titled Large Screen Display LCD or CRT are still not easy to be made as large as

a wall.

Titling multiple projector images into a large screen display is flexible and relatively cheap.

The shortcoming is we need to align the images from different projectors in both geometry and color.

Manually adjusting the projectors is a tedious task.

Camera project system can be used to solve the problem.


Titled Images

Projector images

The camera view

CalibrationPatterns(Projector to Camera Homography can be computed based on these patterns)

1

2 3 4

56 7 8

910

1112


Large Screen Projection

Projector images

The camera view

1

2 3 4

56 7 8

910

1112

Thebigimage

pcamera = Hqprojectorp

q

3

projector image


Edge and Color Alignment

Colors of different projectors are usually different. A color calibration and adjustment procedure is needed.

Blending

2 3

Color(p) = Color(2,p) + (1-)Color(3,p)

p

is determined by the dominance of 2 or 3.


Blending

The size of the blending region has to be carefully selected

If the region is too small, large scale structures will show abrupt changes. If it is too big, small structures (edges, dots) will overlap in a big region and therefore will result in blur (ghost) images.

1-

Blending region


Multi-band Blending

Multi-band blending can be used to address the problem.

In multi-band blending, – images are filtered into different bands. – The mask is also low-pass filtered to generate mask for

each band images.– Images are blending in each channel.– The blended images are summed up to get the final result.


Image Pyramids

The Gaussian Pyramid


Image Pyramids

The Laplacian Pyramid


Image Pyramids

The Gaussian pyramid of the mask


Multi-band Blending

Multi-band blending


Multi-band Blending


System Issues of Large Screen Display

Large screen projection needs multiple projectors working simultaneously.– The first structure uses share memory system such as SGI

Oynx2, that employs a shared-memory model. A single program can have different threads writing OpenGL primitives into different pipes while reading from a single shared database and synchronizing display update over shared flags.

– The second structure is PC cluster, in which each PC handles one projector. This framework is much cheaper but the synchronization is a challenge problem.


Applications

Visualization and Collaboration

IEEE Computer Graphics and Applications, 2000


Immersive Reality

The CAVE (University of Illinois at Chicago)

IEEE Computer Graphics and Applications, 2000


Projection for Augmented Environment

Anton Treskunov and Jarrell Pair, PROJECTOR-CAMERA SYSTEMS FOR IMMERSIVE TRAINING, ASC06


Input Methods for Large Screens

It presents challenges for traditional input methods.

Possible HCI methods include:– Pointing devices, including 3D tracking, passive

optical (video) tracking, ultrasonic tracking, mice, and tablet interfaces;

– User tracking, for point-of-view rendering or for gaze directed interaction, via optical tracking or electromagnetic tracking;

– Handheld devices, providing control interfaces that can be out of band from the display;

– Voice commands with audio feedback; – And haptics interfaces.


Multiple Camera System

Multiple camera system can capture video from different locations simultaneously.

Stereo system has been widely used for inferring the “depth” of objects.


3D reconstruction from Multiple Views


Application of Multi-camera Imaging

Image based rendering The Matrix


Cameras in the Matrix


Camera Calibration Toolboxes

OpenCV

Matlab Calibration Toolbox– http://www.vision.caltech.edu/bouguetj/calib_doc/

Multiple Camera Calibration– http://cmp.felk.cvut.cz/~svoboda/SelfCal/

http://www.vision.caltech.edu/bouguetj/calib_doc/

Cameras and Projectors

Documents

Transcript of Cameras and Projectors