Mitsubishi Electric Research Laboratories Raskar May 03 Seamless Multi-Projector Display on Curved...

Mitsubishi Electric Research Laboratories

Raskar May 03

Seamless Multi-Projector Display on

Curved Screens

Jeroen van Bar, Thomas Willwacher, Srinivas Rao, Ramesh RaskarMitsubishi Electric Research Labs

Cambridge, MA USA


Raskar May 03

Curved Screen Displays

Multiple overlapping projectors on curved screens

Goal :• Replace single-proj• Higher resoltn/brightness• Sub-pixel auto-alignment• Parametric solution• Low cost infrastructure

Markets

Planetarium

Curved screens


Raskar May 03

Dome Projection TechniquesDome Projection Techniques

Edge-Blended (Tiled/Mosaic) Display

Edge-Blended (Tiled/Mosaic) Display

• Sub-Frames w/Spherical Mapping & Edge-Blends• Sub-Frames w/Spherical Mapping & Edge-Blends


Raskar May 03

2

1

3 4

Dome Scree

n


Raskar May 03

OutlineSeamless Curved Display

• Multi-projector low cost method

• Simplified Quadric Image Transfer

• Calibration with camera-pair

• Parametric Rendering solution


Raskar May 03

Related Work

• Conventional Displays– Manual alignment, expensive infrastructure

– [Jupiter,Trimensions, CAVE, Planetaria, Flight Simulators]

• Planar Screens– Camera in loop, auto calibration, low cost– Exploit homography parameters

– [Raskar98,Surati99,Chen00,Brown02 ……]

• Curved Screens– Non-parametric solutions

– [Jarvis97,Raskar98,Yang01 …]

– Parametric– ?, Siggraph 2003


Raskar May 03

Parametric Approach

• Advantages– Lower camera resolution– Tolerance for pixel localization errors– Faster calibration– Efficient well-defined warping

• Avoid look up tables


Raskar May 03

Parametric Image Transfer

X

i

j

Planar Homography Quadric Transfer

X

i j

eiEiT iAj x 33 iAj x 33


Raskar May 03

Planar projective transfer What is homography ?

– Two images of 3D points on a plane are related by a 3x3 matrix

M

i

j

Proj 1

Proj 2

j = A3 x 3 i


Raskar May 03

a1 a2 a3

b1 b2 b3

c1 c2 c3

What is homography ?

Two images of 3D points on a plane

Related by a 3x3 matrix

Proj 1

A3 x 3

Proj 2

i

j

jx

jy

1

j = A3 x 3 i~

=

ix

iy

1

k

jx = (a • i) / (c • i)

jy = (b • i) / (c • i)


Raskar May 03

Current Multi-Cube System MERL Projector Planar Mosaic

Feature Comparison

Many hours Time to install Few minutes

$20,000/projector cube Cost $5,000 per projector

15 kg/projector cube + heavy support

Weight3 kg/projector, no fixed support

structure

Many hours Alignment 10 seconds !

Needs regular care Maintenance Very low

Small gaps Image quality Seamless !, Subpixel alignment

Only high end: command and control, advertising

MarketMiddle segment: Shops,

schools, offices, trade shows

1

12

2

3

3

4

4

Planar Displays


Raskar May 03

Curved ScreensView for a Sweet-spot

Projector

Sweet spot

(Static user)


Raskar May 03

Calibration for a Sweet-spot

Camera at Sweet spot

Projector


Raskar May 03

Discretized non-parametric approach

Projector Image

Camera Image =

p1

c1

c6

p6

Desired Image

Camera at Sweet spot

Projector


Raskar May 03

• Off-Axis Spherical Distortion• Off-Axis Spherical Distortion

Ideal ViewpointIdeal Viewpoint Offset ViewpointOffset Viewpoint


Raskar May 03

Fish-eye ProjectionPlanetaria and Digital Dome Theaters

Fish-eye ProjectionPlanetaria and Digital Dome Theaters

• Immersive Production Software– Spitz - PolyDome™

– SkySkan - DigiDome™

• Immersive Production Software– Spitz - PolyDome™

– SkySkan - DigiDome™


Raskar May 03







Raskar May 03

Curved projective transfer Quadric classification

Projectively equivalent to sphere:

Ruled quadrics:

hyperboloids of one sheet

hyperboloid of two sheets

paraboloidsphere ellipsoid

Degenerate ruled quadrics:

cone two planes


Raskar May 03

Quadrics

Q : 4x4 symmetric matrix,

0QXX T

Q

For 3D points X on Quadric

Nine d.o.f

In general 9 points in 3D define quadric

X

Q


Raskar May 03

Quadric Image Transfer

• Quadratic image transfer function

exQxxqxqBxx TTT ))((' 332

133Tq

qQQQuadric written as

21 params, 4 more than necessary !

X

x x’

[Shashua97]


Raskar May 03

Simplified Quadric Image Transfer

Our Solution

exQxxqxqBxx TTT ))((' 332

eExxAxx T

'

33QqqEeqBA TT

Based on observation ..

X

x x’

17 param warp


Raskar May 03

Simplified Quadric Image Transfer

eExxAxx T

'X

x x’17 param warp

Planar homography: 4 corresponding pixelsQuadric transfer: 9 corresponding pixels


Raskar May 03







Raskar May 03

Calibration of Quadric Screens


Raskar May 03

Approach

Calibration• At each projector i,

– Project structured pattern– View with stereo camera– Finding camera to projector quadric transfer,

Run-time• At each projector i,

– Pre-warp input image using

i

i


Raskar May 03

Low-res Camera 640x480 images

But each Projector 1024x768

CalibrationFinding relationship between camera and projector


Raskar May 03

Non-linear Refinement

Linear EstimationError ~10 pixels

NonLinear RefinementError ~ 1.0 pixels


Raskar May 03Before Blending


Raskar May 03After Blending


Raskar May 03

Projector Framebuffers

Intensity Correction in Overlap


Raskar May 03Projector Framebuffers


Raskar May 03Projector Framebuffer Intensity Weights


Raskar May 03







Raskar May 03

Rendering a 3D Scene

Steps at each projector(Pre-distort vertex 3D location)

• For each triangle T with vertices {Mj}– For each vertex M

• Find pixel m via VirtualViewProjection( M )

• Find warped pixel m’ via quadricTransfer of m

• Replace M with m’


Raskar May 03

Vertex Shader for Quadric Transfer in Cg

vertout main( appin IN, uniform float4x4 modelViewProj, uniform float4 constColor, uniform float3x3 A, uniform float3x3 E, uniform float3 e) {vertout OUT;float4 m1 = float4(IN.position.x, IN.position.y, IN.position.z, 1.0f );float4 m, mi ; float3 m2,mp; float scale;

m = mul( modelViewProj, m1);m2.x = m.x/m.w; m2.y = m.y/m.w; m2.z = 1;scale = mul(m2, mul(E,m2));mp = mul(A,m2) + sqrt(scale)*e;mi.x = m.w * (mp.x)/(mp.z);mi.y = m.w * (mp.y)/(mp.z);mi.zw = m.zw;OUT.position = mi; OUT.color0 = IN.color0; // Use the original per-vertex color specified

return OUT;}

eExxAxx T

'

ParametricWarp


Raskar May 03

Rendering 2D + 3D scene

Concave Dome Convex Dome


Raskar May 03

Details I Skipped ..

• Estimating camera and projector params– Internal and External params– Issue with near-planar 3D points

• Finding pixels weights for blending

• Non-linear optimization

• Rendering– Warping and Depth buffer issues


Raskar May 03

Seamless Curved Display



• Complete Parametric calib+rendering solution

More info : www.raskar.com/Projector/


Raskar May 03

Projector Mailing List

[email protected] projector

Projector bibliography

www.raskar.com/Projector/


Raskar May 03

Advantages

• Parametric warp– Lower camera resolution– Tolerance for pixel localization errors– Faster calibration– Efficient well-defined warping

Mitsubishi Electric Research Laboratories Raskar May 03 Seamless Multi-Projector Display on Curved...

Documents

Transcript of Mitsubishi Electric Research Laboratories Raskar May 03 Seamless Multi-Projector Display on Curved...