Efficient stereoscopic and 3D visualization on …aht/VizPhD2018/Stereo.pdfParallax: Viewing on...

Efficient stereoscopic and 3D

visualization on planar displays

Stefan Seipel

2018-11-27

Omnipresence of stereo 3D

Movies Industry pushing

for modern 3D technology

Consumer electronics

industry pushes 3D TV

Need your own content?

You need a 3D camera!

You got a 3D camera?

Need a 3D picture frame!

You’ve got all this?

Buy a 3D mobile phone!

More recently:

Simultaneous localization and mapping (SLAM) techniques are entering

the scene.

Mainly used for navigation

unmanned aerial vehicles (UAV)

self-driving cars and motorbikes

other autonomous vehicles

Just recently also used for positioning in Augmented Reality

e.g. Microsoft HoloLens

Google Project Tango

=> Triggers a new wave of 3D content production/consumption

Early adopters of modern stereoscopy

G.R.A.P.H. TableUppsala Univ. (1999)

3D Table at HiGGraphiX Center (2002)

Computer Sweden, 20th of October 20043D GIS, GraphiX Center, 2011

How does it work?

Human stereoscopic vision

Some available techniques for stereo 3D

How can we best recreate stereographic images?

Learning from other fields helps creating great content

When converging on an object at some

certain distance, a point at some different

distance in the scene will appear on

the retina with some horizontal disparity.

Stereopsis: Process of merging two retinal

images into one 3D image by evaluating

horizontal disparities.

Retinal disparity (retinal image)

vs.

Stereo parallax (stereographic image)

Viewing in the real 3D world

IPD

Ve

rge

nce

Dis

tan

ce

1

2

left ”retinal image” right ”retinal image”

The perceptual basis for stereoscopic displays

How small a difference in depth can we see?

Stereo acuity

• Ability to resolve objects in depth (based on stereo)

This is intrinsically limited by retinal resolution

• Angle of binocular disparity dγ

• Measured as the difference of two

angles for a just detectable difference (radians)

• Typically 0.5 minute of arc (0.000144 rad)

• At 6 meters viewing distance dz~8cm

• At 25 cm meters viewing distance dz~0.014 cm

Perceptual issues of stereoscopic displays

Viewing virtual 3D images on a 2D screen is different

from natural 3D viewing.

All pixels have actually the same distance to viewer!


Parallax: Viewing on plano-stereoscopic displays

Parallax is lateral displacement of homologous points on the planar display.

Zero Parallax

Converging at a point on a

display surface causes homologous

points on display to have zero parallax.


Positive Parallax

Converging at a virtual point behind the display

surface causes homologous points on display

to have positive parallax.

This point is said to be in screen space.




Negative Parallax

Converging at a virtual point in front of the display

Surface causes homologous points on display to have

negative parallax.

This point is said to be in viewer space.




Converging the eyes’ axes on a real point in 3D space implies verging the eyes towards that point.

The neuro-muscular functions to control (con)vergence are also evaluated to assess distance.

The relationship between convergence and accommodation in natural viewing

The point is kept in focus by adjusting the

lens (accommodation).

Note: Accommodation is a monocular cue!

Under normal natural viewing conditions, accommodation and convergence correspond.

The correspondence of convergence and accommodation is habitual an can be voluntarily

put out of function (crossing the eyes).

One dilemma of stereo-graphic 3D images

The accommodation – convergence conflict (stereo-graphic visualization)

Keeping homologous points on screen in focus

requires accommodation at screen distance Ds.

DsIn natural viewing of the real world accommodation

and convergence are coupled processes.

But:

Seeing plano-stereoscopic 3D images correctly, requires

accommodation and convergence on different distances.

Pixels on screen have ”fixed” distance to viewer

Accommodation is a neuromuscular function of the

ciliari muscles controlling the lens.

Convergence is a occulo-motoric function.

Dc

To support fusion of homologous points, the eyes

must converge at Dc – the distance to the

virtual point.

Avoiding AC-Conflict

In practice, comfortable stereo graphics is a trade-off

Stereoscopic Comfort Zone

Comfort

VisualDiscomfort

VisualDiscomfort

Screen

Fixed Variables:

Focal distance to screen

Maximum tolerable parallax

larger depths in screen space

limited depths in viewer space


Summary:

• Regions in a stereographic image with some parallax give rise to retinal disparity,

which is one cue for perceiving depth in a scene.

• Stereo parallax and retinal disparity are related

• The brain can fuse binocular retinal images into one spatial image (stereopsis)

• Equivalently, there are limits to display parallax.

(e.g. 1,5° of visual angel Lipton[1991])

• Viewing non-zero parallax images on plano-stereoscopic displays compromises

the accommodation/convergence relationships developed for/in natural viewing.

*) more at http://paulbourke.net/stereographics/stereorender/

Practical issues in stereo graphics

Lipton [1991] / Bourke[1999] – Parallax Control

Don’t exceed parallax values of more than 1.5° visual angle [Lipton] *)

(rather than using explicit parallax values on screen).

R

L

1.5°

Viewing distance D

On-screen parallax osp

D [cm] 50 75 100 200 300 400

osp [cm] 1,31 1,96 2,62 5,24 7,86 10,47

Practical examples for on-screen parallax values:

*) Bourke recommends osp/D < 1/30 (~1.9°)


Virtual spatial depth due to observer distance and limited parallax.

Negative parallax situation:Allowable spatial depth d towards viewer space

IPD

Vie

win

g D

ista

nce

D

1

...

osp

IPD

Dd

dD

d

IPD

osp

d

osp

Example for IPD = 6,5 cm

D [cm] 50,00 75,00 100,00 200,00 300,00 400,00

osp [cm] 1,31 1,96 2,62 5,24 7,86 10,47

d 8,38 17,40 28,72 89,24 164,17 246,83

d/D 0,17 0,23 0,29 0,45 0,55 0,62


Virtual spatial depth due to observer distance and limited parallax.

Positive parallax situation:Allowable spatial depth d into screen space

IPD

Vie

win

g D

ista

nce

D

1

...

osp

IPD

Dd

dD

d

IPD

osp d

osp

Example for IPD = 6,5 cm

D [cm] 50,00 75,00 100,00 200,00 220,00 245,00

osp [cm] 1,31 1,96 2,62 5,24 5,76 6,42

d 12,61 32,47 67,47 829,45 1714,79 18612,49

d/D 0,25 0,43 0,67 4,15 7,79 75,97


Remember:

Values sometimes seen as a rule of thumb/best practice in specific

situations. (Lipton’s seem to work for many VR applications)

Values do not generally apply for all viewing conditions

(Desktop Viewing, Head-Mounted Displays, Cinema have different

convergence distances)

Once you know the basics, composing stereoscopic

images is an art, not a science. (Lenny Lipton,1991)


Does my desktop 3D application also work in the VR theatre?

Many visualization applications are parameterized to work in stereo on desktop systems.

Example:

17” screen approx. 34 cm wide.

Application creates maximum parallax of 2 cm, which is comfortable when

viewed from approx. 70-80 cm distance.

You bring your application to the computer in the VR theatre, without altering viewing parameters.

The stereo-picture is now blown up to 3 meters horizontal size i.e. 850%.

Maximum parallax on the projection screen is now 17,14 cm.

According to Lipton’s recommendation for parallax of max. 1.5°, the observer should be seated

at least 6,6 meters from the screen!

Note: Viewing parameters for stereo-viewing do NOT scale linearly!!!

A general problem in computer graphics

Viewing metaphor in use is not suitable for VR and stereo 3D!

Traditionally the field of computer graphics has adopted the metaphor of

a synthetic pin-hole camera

Projection math is then

* based on central perspective projections

* adapted to parameters such as horizontal and vertical field of view

… even worse: focal length and aspect ratio

This is manifested in industry standard APIs (e.g. OpenGL/GLUT, ”gluPerspective”)

e.g. standard stereo render mode in vtk is based on central perspective projections


What is wrong with a camera model?

1. The scale level in recording/synthesizing an image is not related to the spatial

relations between observer and display during presentation.

This is usually not a problem in monoscopic graphics, but in stereo it is!

(parallax and convergence issues, discussed before)

Example:

Hyper-stereo and excessive parallax when photographing a frog from 30 cm

distance with a IPD of 6 cm and display on a 4m x 3m screen.

What focal length is applicable for visualizing atoms and molecules?


What is wrong with a synthetic camera model?

2. Binocular viewing of (projected) plano-stereoscopic displays requires off-axis

projected images

typical “toe-in”

camera setup

For stereo

Individual, non co-planar

projection planes

Same display plane

when viewing both images

Left eye Right eye




projected images

Projection images on individual

focal planes

Left eye Right eye

Projection images as apparent on

display plane

Apparent virtual object

Real object

typical “toe-in”

camera setup

For stereo




projected images

Co-planar projection planes ->

require off-axis projections

Left eye Right eye

Real object

Apparent virtual object

Required camera

setup for stereo

Stereo image capture

co-planar image planes


What is wrong with a camera model?

3. A camera model with central perspective projection assumes the

viewer to be located in front of the center of the image

”Correct” viewing position”Incorrect” viewing position

Single view with centric position

What can we do about it?

Presentation of 3D objects on planar 2D displays requires generally

an arbitrary relationship between the projection (=presentation) plane

and the center of projection (=observer´s eye).

This viewing metaphor is called ”Window-on-world”, short WoW

or “Fish-Tank Virtual Reality”

A WoW assumes generally an off-axis projection

The result of off-axis projection is an anamorphic image which

,viewed from the CoP, appears geometrically undistorted.

Anamorphic images are not only compelling in computer graphics…

http://www.julianbeever.net/

Works of the famous artist Julian Beever

This is the center of projection

Works of the famous artist Julian Beever


Here is what you see from

the center of projection


[Accessed September 2016, http://distractify.com/old-school/2015/02/22/mind-blowing-optical-illusions-1197825445]

How can I accomplish good off-axis projections?

A metrically correct Window-on-world projection requires

a) the relation between window and observer position to be

modeled at the same level of scale (same WCS)

b) requires a parameterization of an appropriate off-axis

projection matrix

c) zooming of objects in the scene is accomplished through

manipulation of object parameters rather than viewing parameters

A) and b) usually require that you implement the projection pipeline

into your Vis-App yourself to be sure that the job gets done right.

How to parameterize an off-axis projection matrix?

nearfar

nearfarD

nearfar

nearfarC

bottomtop

bottomtopB

leftright

leftrightA

2

glFrustum(left,right,bottom,top,near,far);

Using built-in OpenGL frustum function

0100

000

02

0

002

C

Bbottomtop

near

Aleftright

near

P

(observer at origo, facing towards –z, near defines proj. plane)

left

righttop

bottom

near

Parameterize glFrustum for arbitrary WoW


window position

width

height

up-vector

side-vector

viewer position

A window on world can be arbitrarily identified by:

Window position: wp

Orientation vectors (roll/tilt): sv, up

Width and height: w, h

And an arbitrary observer position: vp



wp

w

h

up

sv

vp

A window on world can be arbitrarily identified by:

Window position: wp

Orientation vectors (roll/tilt): sv, up

Width and height: w, h

And an arbitrary observer position: vp

This WindowOnWorld is part of a

central perspective projection along

(cp-vp)

Which is frustrated at

left, right, top, and bottom

cp

left

top

bottom

right



wp

w

h

up

sv

vp

Calculating glFrustum parameters from WoW parameters:

cp

left

top

bottom

right

upsvwn 0,10.1 upsv

wpvpvv

vv

wn

D

wnvvD

wnDvpcp



wp

w

h

up

sv

vp


cp

left

top

bottom

right

upsvwn 0,10.1 upsv

wpvpvv

vv

wn

D

wnvvD

wnDvpcp bl tr

cpupsvwptr

cpupsvwpbl

5.05.0

5.05.0



wp

w

h

up

sv


cp

left

top

bottom

right

upsvwn 0,10.1 upsv

wpvpvv wn

wnvvD

wnDvpcp bl tr

uptrtop

svtrright

uplbbottom

svlbleft


Adjusting frustum parameters for clipping plane

viewer position

left

top

bottom’

right

cp

D

bottomright’

top’

left’

cd

glFrustum assumes clipping plane

To co-incide with plane of projection

D

cdDtoppto

D

cdDrighttrigh

D

cdDbottommbotto

D

cdDlefttlef

cdDnearHence :


Pre-multiplying modelview matrix

viewer position

left

top

bottom’

right

cp

D

bottomright’

top’

left’

cd

glFrustum assumes further:

vp is at origin

(cp-vp) pointing toward -z

1

1000

vpwnupsvPM

Translating vp to O and frame alignment

Be pre-loading model-view matrix PM

glMatrixMode(GL_PROJECTION);

glLoadIdentity(),

glFrustum(left’,right’,bottom’,top’,near,far);

glMatrixMode(GL_MODELVIEW);

glLoadIdentity();

Gl(MultMatrixd(PM);

Putting it together

Inconsistent interaction between

3D content and screen surround

destroys illusion:

Virtual 3D object intersects with

screen borders.

Contradictious spatial cues

Stereo-disparity says:

“object is in front of screen”

Object occlusion by surround says:

“object is behind the screen”

Sabotaging a good 3D illusion

Giving to much freedom to the user when interacting…

Sabotaging a good 3D illusion

…can cause hyper-parallax situations leading to diplopia

Example of dynamic perspective conditions

First person view with dynamic perspectiveAlternate observer’s percept

Summary: Cues for 3D image generation?

Stereopsis (and correct stereo) only one of many spatial cues!

Correct parallax (software)

Projector adjustment/image alignment (hardware)

Scaling conditions (sw + hw)

Interference with surround

Utilize shadows, shading

Exploit motion parallax

Object rotation creates strong optic flow

Keep noise or small spatial features/structures in e.g. 3D imagesIf objects with lack of spatial detail, then use textures

Try to use dynamic perspective conditions

3D Display Categories

True Volumetric

Displays

Stereographic Displays

(2 projected planar views)

Single Display Plane

Multiplexed

Dual Display

Immediate Mode

Temporal MUX Spatial MUX Chromatic MUX

+

Spectral MUX

Polarization MUX

Time Multiplexed Stereo Image Pair

V-Sync at 60 Hz

Addidional V-Sync at 120 Hz

(enforced with sync. doubler)

op

en

clo

se

clo

se

op

en

Active Shutter Glasses

(LCD-Shutters)

Temporal Multiplexing and “Ghosting”

0%

100%

0%

100%

Shutter Opacity Left Eye

Shutter Opacity Right Eye

Pixel Intensity (Phosphor)

Open Right

Close Left

Open Right

Close Left

Open Right

Close Left

Close Right

Open Left

Close Right

Open Left

Two Pixels are Drawn On-Screen:

Purple Pixel for the Right Eye

Green Pixel for the Left Eye

120Hz screen refresh rate

t

t

t8.3 ms

Left eye sees purple pixel due to after-glowing

Right eye sees green pixel due to shutter response

Color Encoded Stereo Image Pair

Chromatic Multiplexing - Anaglyph Technique

Images curtsey: http://axon.physik.uni-bremen.de/research/stereo/color_anaglyph/index.html#ana

Encoding:

A stereo image pair is combined

into one so called anaglyph.

The left eyes view is encoded

with the red colour component.

The right eye view is encoded with

the complementary colour i.e. green

And blue colour components.

Decoding:

The left eye uses a red colour filter

that passes through dominant red

components.

The right eye uses a cyan colour filter

that passes through dominant blue

And green components.

Chromatic Multiplexing - Anaglyph Technique

Conventional filter materials

are used to separate the color

spectra of a stereogram.

Image splitting by chromatic

separation.

spectrum of the red filter

spectrum of the cyan filter

Spectral Filtering – Infitec Teknology

More info at: http://www.infitec.net/infitec.pdf

Narrow band spectral filters allow different

sub-bands of the tri-stimulus spectra.

Advantage: Filterbased optical multiplexing

which provides ”natural color” stereo image

separation.

Left eye

Spectral filter

Right eye

Spectral filter

Spatial Multiplexing

VR-3100

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L R

R L

L

R

L

R

L

R

L

R

L R L R L R L R

The stereo image pair is presented on alternating

pixels, pixel columns, or scan-lines.

Optical arrangements are used to block-out

the the wrong image for the corresponding eye.

Effective resolution is reduced.

Stereo-rendering in vtk

Classes involved in stereo rendering in vtk:

vtkRenderWindow- contains a flag that indicates stereo-rendering

- responsible for the output of stereo image pairs on screen

- offers a variety of image multiplexing modes

vtkRenderer- contains a flag that indicates stereo-rendering

- manages rendering several passes

- does not contain any else relevant information for stereo rendering

vtkCamera

- contains actual viewing and projection matrices

- standard built-in support for stereo viewing

- allows ”digging deeper” , internals poorly documented

Example of methods to control stereo parameters in vtk:

vtk:RenderWindow

::StereoCapableWindowOn()

::StereoRenderOn()

::SetStereoType()

::SetStereoTypeToAnaglyph()

::SetStereoTypeToInterlaced()

::SetStereoTypeToCheckerboard()

::SetStereoTypeToCrystalEyes()

vtkCamera

::stereo ”flag indicating stereo rendering”

::SetEyeAngle(double) ”set separation between eyes in degrees”

::SetUseOffAxisProjection(int) ”off axis viewing frustum is used”

::ComputeOffAxisProjectionFrustum()

::SetEyeSeparation(double) ”used for OA projections, default 0.06”

…



# This program demonstrates how VTK can be used

# to render a text

# The user can also interact with the text by

# using the mouse

# load VTK

import vtk

# Create a Text source and set the text

text = vtk.vtkTextSource()

text.SetText("UPPMAX")

text.SetForegroundColor(0.6,0.2,0.2)

# Create a mapper and set the Text source as

# input

textMapper = vtk.vtkPolyDataMapper()

textMapper.SetInputConnection(text.GetOutputPor

t())

# Create an actor and set the mapper as input

textActor = vtk.vtkActor()

textActor.SetMapper(textMapper)

# Create a renderer

ren = vtk.vtkRenderer()

# Assign the actor to the renderer

ren.AddActor(textActor)

# Create a rendering window

renWin = vtk.vtkRenderWindow()

# Add the renderer to the window

renWin.AddRenderer(ren)

# Set the name of the window (this is optional)

renWin.SetWindowName("Hello World!")

# Enable stereo rendering

renWin.StereoCapableWindowOn()

renWin.SetStereoTypeToInterlaced()

renWin.StereoRenderOn()

# Make sure that we can interact with the

# application

iren = vtk.vtkRenderWindowInteractor()

iren.SetRenderWindow(renWin)

# Initialze and start the application

iren.Initialize()

iren.Start()

Some references

Additional reading for this lecture:

http://www.cs.unc.edu/Research/stc/FAQs/Stereo/stereo-handbook.pdf

Research community in stereoscopic/3D displays

http://www.stereoscopic.org/2019/index.html

http://www.cs.unc.edu/Research/stc/FAQs/Stereo/stereo-handbook.pdf



Efficient stereoscopic and 3D visualization on …aht/VizPhD2018/Stereo.pdfParallax: Viewing on...

Documents

Transcript of Efficient stereoscopic and 3D visualization on …aht/VizPhD2018/Stereo.pdfParallax: Viewing on...