Virtual Reality ll

Visual Imaging in the Electronic Age

Donald P. GreenbergNovember 16, 2017

Lecture #22

Fundamentals of Human Perception• Retina, Rods & Cones, Physiology

• Receptive Fields

• Field of View

• Visual Acuity of Resolution

• Opponent Color Theory

• Compression

• Bandwidth Limitations

• Saccades

The Optomotor Cycle

Extraocular Muscles

“Foundations of Sensation and Perception.” Mather, George. 2009.

Saccade Control• Saccade control is the ability of the eye(s) to move quickly from one fixation

point to another (100-300 ms)

• To obtain a complete picture, normal adults perform 3-5 saccades (“snapshots”) per second

• Fixation “restops” are ≈ 50-100 ms

The eye jumps, comes to rest momentarily (producing a small dot on the record), then jumps to a new locus of interest.

- David H. Hubel. EYE, BRAIN, AND VISION, 1988 Scientific American Books, Inc. p. 80.

Saccadic Motion

Saccadic Masking• Visual saccadic suppression

• The brain selectively blocks visual processing during eye movements

• Neither the motion of the eye or subsequent motion blur of the image nor the time gap in visual perception is noticeable to the viewer

Saccadic Masking• There are two major types of saccadic masking or suppression

• Flash suppression is the inability of the light to see a flash of light during a saccade

• Suppression of image displacement is characterized by the inability to perceive whether a target has moved during a saccade.

Peak Angular Velocity

Wikipedia

Human Depth Perception

Depth Perception

Oculomotor

Binocular

Convergence

Monocular

Accommodation

Visual

Binocular

Stereopsis

Monocular

Static Cues

PerspectiveFamiliarity,

Relative Size

Motion, Position Occlusion Texture

GradientShading, Shadows, Highlights

Atmospheric Blur

Motion Parallax

Monocular Human Depth Perception

Depth Perception

Oculomotor

Binocular

Convergence

Monocular

Accommodation

Visual

Binocular

Stereopsis

Monocular

Static Cues

PerspectiveFamiliarity,

Relative Size

Motion, Position Occlusion Texture

GradientShading, Shadows, Highlights

Atmospheric Blur

Motion Parallax

Monoscopic Depth Cues• Perspective

• Depth from Motion, Relative Size, Position, Familiarity

• Occlusion

• Texture Gradient

• Parallax from Motion

• Shadows and Specular Highlights

• Atmospheric Blur

Monoscopic Depth Cues• Perspective

• Depth from Motion, Relative Size, Position, Familiarity

• Occlusion

• Texture Gradient

• Parallax from Motion

• Shadows and SpecularHighlights

• Atmospheric Blur

Monoscopic Depth Cues• Perspective

• Depth from Motion, Relative Size, Position, Familiarity

• Occlusion

• Texture Gradient

• Parallax from Motion

• Shadows and SpecularHighlights

• Atmospheric Blur

Monoscopic Depth Cues• Perspective

• Depth from Motion, Relative Size, Position, Familiarity

• Occlusion

• Texture Gradient

• Parallax from Motion

• Shadows and SpecularHighlights

• Atmospheric Blur

Monoscopic Depth Cues• Perspective

• Depth from Motion, Relative Size, Position, Familiarity

• Occlusion

• Texture Gradient

• Parallax from Motion

• Shading, Shadows, and Specular Highlights

• Atmospheric Blur Viewpoint A Viewpoint B Viewpoint C

Monoscopic Depth Cues• Perspective

• Depth from Motion, Relative Size, Position, Familiarity

• Occlusion

• Texture Gradient

• Parallax from Motion

• Shading, Shadows, and Specular Highlights

• Atmospheric Blur

Monoscopic Depth Cues• Perspective

• Depth from Motion, Relative Size, Position, Familiarity

• Occlusion

• Texture Gradient

• Parallax from Motion

• Shading, Shadows, and Specular Highlights

• Atmospheric Blur

Monoscopic Depth Cues• Perspective

• Depth from Motion, Relative Size, Position, Familiarity

• Occlusion

• Texture Gradient

• Parallax from Motion

• Shadows and SpecularHighlights

• Atmospheric Blur

Monoscopic Depth Cues• Perspective

• Depth from Motion, Relative Size, Position, Familiarity

• Occlusion

• Texture Gradient

• Parallax from Motion

• Shadows and SpecularHighlights

• Atmospheric Blur

• AccommodationNote change in lens shape

Accommodation

• This is the process by which the vertebrate eye changes optical power to maintain a clear image or focus on an object as its distance varies.

Accommodation

The reflex can be controlled but cannot be ‘felt’Accommodation amplitude declines with age

Vergence

• The simultaneous movement of the pupils of the eyes toward or away from one another during focusing.

• This measure of the convergence or divergence of a pair of light rays is defined as vergence.

Vergence Accommodation Conflict

Human Depth Perception

Depth Perception

Oculomotor

Binocular

Convergence

Monocular

Accommodation

Visual

Binocular

Stereopsis

Monocular

Static Cues

PerspectiveFamiliarity,

Relative Size

Motion, Position Occlusion Texture

GradientShading, Shadows, Highlights

Atmospheric Blur

Motion Parallax

Binocular Vision

• Binocular Vision, which is the basis for stereopsis is important for depth perception and covers 114 degrees(horizontally) of the human visual field.

• The remaining sixty to seventy degrees have no binocular vision (because only one eye can see those portions of thevisual field)

Stereoscopic Depth Cues

• Stereopsis

– Horizontal Parallax– Occlusion Revelation

• Shape Change

• Convergence

Stereoscopic Depth Cues

• Stereopsis

• Shape Change

– Standard Stereo– HypoStereo (Gigantism)– HyperStereo (Dwarfism)

• Convergence

Standard

HypoStereo

HyperStereo

Stereoscopic Depth Cues

• Stereopsis

• Shape Change

• Convergence

– Maintain single binocular vision

– Fusion

“Vision and Visual Disabilities – An Introduction,” by Gerd Waloszek, SAP User Experience

Moore’s Law

“Chip density doubles every 18 months.”

Processing Power (P) in 15 years:

Exponential Laws of Computing Growth

War for the Planet of the Apes

Off-line/On-line 𝐑𝐑𝐑𝐑𝐑𝐑𝐑𝐑𝐑𝐑

10 ℎ𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜10𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑜𝑜𝑚𝑚𝑚𝑚𝑜𝑜𝑚𝑚𝑚𝑚𝑜𝑜

= (10 ℎ𝑜𝑜𝑜𝑜.)(60𝑚𝑚𝑚𝑚𝑚𝑚

ℎ𝑟𝑟 )(60 𝑠𝑠𝑠𝑠𝑠𝑠𝑚𝑚𝑚𝑚𝑚𝑚)(1,000𝑚𝑚𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠)

10𝑚𝑚𝑜𝑜

= 3.6 × 106/eye

= 7.2 × 106

Potential Improvements

• Eye tracking and foveal rendering

• Multi resolution displays

• Monoscopic vs. stereoscopic level of detail

• Asynchronous spacewarp

Increasing Densities (ppi) of OLED Displays

Field of View of the Human Eye

Wikipedia

Foveal Eye Tracking Constraints 2016

• Speed- needs to be fast enough to meet update requirements (currently 11 milliseconds, 90 Hz)

• Accuracy- Gaze direction is < 0.5 degree• Foveal direction accuracy can be ~ 1.0 arc minutes (1/60 of a degree)

• Non-invasive measurements- still need to see entire visual field

Eye Tracking

Sensing Methods: Retinal Tracking

● Hard problem with current technologies○ Extremely difficult to illuminate

■ Must bounce light off of retina■ Light comes back through iris

○ Light must be extremely bright○ Too much exposure will damage retina

● Typically done in ophthalmological setting

● Presently can only detect faint images ofblood vessels, companies working on it

● Very high angular resolution, but would presently require occlusion of vision

Purkinje Reflections

Purkinje Reflections

Cornsweet and Crane 1973

Purkinje Reflections

1st and 4th Purkinje Reflections

No Foveated Rendering

Roadtovr.com

Roadtovr.com

Contemporary ‘blur’ foveated rendering

Roadtovr.com

NVIDIA’s ‘contrast preserving’ rendering

Research on Foveated Displays

Alaskan Moose Diorama

Dall Sheep Restoration

Alaska Brown Bear Diorama

Plan of Typical Diorama

LOD Image Based Primitives

Layered Depth Images

Space & Time Warping

• When frame rates are not met, there are several types of solutions, but all have their deficiencies

• Visual artifacts appear because of loss of accuracy– e.g. Imperfect extrapolation, Object Disocclusion trails

Asynchronous Spacewarp

Asynchronous Spacewarp

Asynchronous Spacewarp

Asynchronous Spacewarp

Asynchronous Spacewarp

End