Augmented and Virtual RealityMikko Kytö7.11.2017

CS-3120 Human-Computer Interaction

From Real to Virtual

21.11.20172

[1] Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, E77-D(12), 1321–1329.

Augmented Reality (AR)

According to definition: 1. AR application must mix real and virtual2. Registered in 3D with physical world3. Interactive in Real Time

21.11.20173

Why AR?

Enables providing augmented information and guidance in the physical context

Enables investigating virtual 3D models in physical environment

21.11.20174

http://www.nlr.org/article/x-lab-frontrunner-augmented-reality-mro-aerospace/

http://ngm.nationalgeographic.com/img/big-idea/augmented/augmented.jpg

Augmented Reality (AR)

According to definition: 1. AR application must mix real and virtual

• DISPLAY TECHNOLOGIES2. Registered in 3D with physical world

• TRACKING TECHNOLOGIES• CAPTURING TECHNOLOGIES

3. Interactive in Real Time• INTERACTION TECHNOLOGIES

21.11.20175

AR display technologies

Modalities• Visual• Aural• Haptic• Kinaesthetic• Olfactory

21.11.20176

Development of AR Displays

21.11.20177

1990 2000 2010 2020?

http://foonacha.blogspot.fi/2015/06/the-dangers-of-augmented-reality.html

Hand-held displays

21.11.20178

https://binocular.io/insights/augmented-reality-for-retailers-in-furniture-and-interior-design

Wearable displaysTwo approaches1. Optical see-through2. Video see-through

21.11.20179

www.youtube.com/watch?v=_x9462x6A0k

Basic optical principle of near-eye display204 JOURNAL OF DISPLAY TECHNOLOGY, VOL. 2, NO. 3, SEPTEMBER 2006

Fig. 1. (a) Geometry of a magnifier. A magnifier forms a virtual image and the observer’s pupil becomes the exit pupil of the system. The distance from the vertexof the last physical surface of the system to the exit pupil is called the eye relief. (b) Example 45-mm focal length magnifiers. Monocular (left) and biocular (right).Both systems are working with a 30-mm screen diameter. The monocular system has a 40-mm ’exit pupil’ and the biocular system has an 84-mm ’exit pupil’, thelatter being large enough to cover both eyes. (Adapted from Williamson [24]). (Color version available online at http://ieeexplore.ieee.org.)

requirement since the microdisplay typically is too small toview with the unaided eye.

Regardless of complexity (i.e., types or number of surfaces),an optical system can always be characterized by an effectivefocal length along with principal planes and nodal points. Amagnifier forms a virtual image when the object lies inside ofits focal length, as shown in Fig. 1. Distance is the object dis-tance and it is measured from the principal plane . Distance

is the image distance and it is measured from the principalplane . We assume the object and image spaces to reside inair, therefore, the nodal points ( and ) are at the same pointsas the principal planes. The formation of the virtual image is il-lustrated by tracing an off-axis ray through the nodal point anda ray parallel to the optical axis of the system, the intersectionpoint of these rays determine the location and size of the virtualimage. The virtual image formed by a magnifier has the sameorientation as the object—sometimes referred to as erect in theoptics community. In HWDs, the virtual image can be placedwithin about 1 m for near-field tasks or beyond 1 m for far-fieldtasks. Optical distances can also be measured in diopters whichis a reciprocal of the optical distance expressed in meters. Thepupil of the observer becomes the limiting aperture and the exitpupil in a magnifier. Magnifiers are typically designed to accom-modate a range of eye movements such as translations and rota-tions while observing the virtual image formed by the magnifier.The eye motion requirement combined with a desire to achievegood image quality motivates designs more complicated than asingle lens. In terms of image quality, we desire that the pointsin the object map to points in the image, as limited by the size

of the pixel or the resolution of the eye, planes map to planesand that the desired magnification remains constant across theimage. Deviations from these desires are called aberrations andit is the task of the optical designer to minimize them.

As an example of noncollimated single lens magnifier,Coulman and Petrie [113] discuss the design of a binocularmagnifier with a conic surface. They compare the shapes andrelative positions of images formed for each eye through abinocular magnifier with spherical surfaces and another mag-nifier combining a spherical and a conic surface. For each lineof sight of each eye, there will be an astigmatic focus, differentfor the tangential and the sagittal directions. The middle pointbetween the tangential and the sagittal surfaces can be takenas the focus surface, which is not necessarily the convergencesurface. Therefore, the magnifier based on spherical surfacescreates images considerably different for each eye whichrequires the eyes to fuse two completely different images,especially toward the edge, leading to a disturbance of theaccommodation and convergence relation. Rogers and Freemanreoptimized the design given by Couldman and Petrie to havea diameter of 79 mm and having a sixth-order aspheric surfaceand they reported a magnification of 2.3 [112].

An eyepiece, in addition to creating a virtual image for thehuman visual system, forms an exit pupil by imaging the pupilof the system prior to it to the image space. The distance betweenthe edge of the last surface and the exit pupil is referred to as eyeclearance. The distance from the vertex of the last surface tothe exit pupil is called the eye relief. A minimum eye clearancedistance is considered to be 20 mm according to Kocian [17]

21.11.201710

Display comparison

Optical see-through Video see-through+ Better visibility of surrounding real world

+ Video image can be controlled àMatching graphics and video feed

+ Lighter - Perception of real world is distorted- Brightness (current displays difficult to use outdoors)

- Latency

21.11.201711

Social acceptability of wearable displays• Lack of knowledge what the

wearer is doing • ‘Glassholes’

• Still bulky and interfere with face-to-face interaction

• Lighter and less noticeable wearable displays will open a lot new possibilities for AR

21.11.201712

Spatial augmented reality

Using projectors • Augmented Climbing Wall

https://www.youtube.com/watch?v=kwticv9ai_Q

21.11.201713

Future display technologies

• Retinal display• Photons projected into eye• Bright (works also outdoors

in a day light)• Challenge in viewing

frustum• Contact lens display

• Big challenges (e.g., power and resolution) to be solved

21.11.201714

http://inhabitat.com/solar-powered-augmented-contact-lenses-cover-your-eye-with-100s-of-leds/

Microvision Virtual Retinal Display

Augmented Reality (AR)

According to definition: 1. AR application must mix real and virtual

• DISPLAY TECHNOLOGIES2. Registered in 3D with physical world

• TRACKING TECHNOLOGIES• CAPTURING TECHNOLOGIES

3. Interactive in Real Time• INTERACTION TECHNOLOGIES

21.11.201715

Tracking technologies

• Active (sending and receiving)• Mechanical, Magnetic, Ultrasonic• GPS, Wifi, Bluetooth, cell location

• Passive• Inertial sensors (Compass, gyroscope, accelerometer)• Optical (markers or natural features)

• Hybrid• Combination of sensors

21.11.201716

Active tracking

• Mechanical tracking: Measuring joints of mechanical arms

• Magnetic tracking: Measuring current generated from moving coil in magnetic field

• Ultrasonic tracking: Measuring changes of travel time of ultrasonic waves

• Wifi, Bluetooth, GSM cell location: Changes in signal strength

• GPS: Changes in radio signal travel time

21.11.201717

MOVEMENT RANGE INCREASES

Tracking with Inertial measurement units (INU)Accelerometers, gyroscopes (and sometimes magnetometer) combined+ Good for tracking orientation+ Cheap and small

- Bad for tracking absolute position over time- Drift, requires calibration

21.11.201718

Optical tracking (with markers)

21.11.201719

http://www.hitl.washington.edu/artoolkit/documentation/userarwork.htm

Optical tracking (markerless)

Tracking based on local image features (for example SURF-features)

21.11.201720

http://www.vision.ee.ethz.ch/~surf/download.html

Optical tracking (unknown environment)SLAM (Simultaneous Localisation and Mapping)

Derived from robotics: ”Where am I?”

https://www.youtube.com/watch?v=5I5pbSs-yrU

21.11.201721

Inertial and optical tracking combined

Good accuracy• Support by operating

systems• Android: ARCore SDK• iOS: ARKit part of iOS

11

21.11.201722

Scene capturing technologies

Depth sensors (like Kinect) needed for improving interaction between virtual and physical

• Solving what is between image features (e.g., texturelessareas)

• Solving changes in the scene (e.g., changes in occlusion)

21.11.201723http://dx.doi.org/10.1016/j.neucom.2014.12.081.

Augmented Reality (AR)

According to definition: 1. AR application must mix real and virtual

• DISPLAY TECHNOLOGIES2. Registered in 3D with physical world

• TRACKING TECHNOLOGIES• CAPTURING TECHNOLOGIES

3. Interactive in Real Time• INTERACTION TECHNOLOGIES

21.11.201724

Interaction technologies

How do I interact with that thing?

21.11.201725

Interaction technologies

• 3D gestural input• Considered to be natural, but how natural

it is to move your hands in the air?• Can become tiring • Haptic feedback?

• Tangible interaction• Data gloves• Head movements• Eye movements• Voice input

21.11.201726

5DT data gloves: http://metamotion.com/hardware/motion-capture-hardware-gloves-Datagloves.htm

3D gestural input

21.11.201727

Device Distance range (m)

Purpose

Leap 0.2 – 1 m Hand trackingKinect 0.7 – 6 m Skeleton tracking, 3D

reconstructionLaser scanners

1 – approx. 200 m

3D reconstruction

AR experience challengesDisplay technology• Especially in terms of depth

perception, 3D stereoscopic displays used mainly in HMDs

Tracking technology

Capturing technology• Lack of interaction between real

and virtual (e.g., occlustions and shadows)

21.11.201728

Applications so far

• Applications in industry• Maintenance (e.g., X-Ray Visualisation), Design,

Training• Consumer applications

• Games (e.g., Pokemon GO), Social media• Navigation, AR Browsers (Layar, Wikitude,..),

language• Interior design• Advertising

• Education (e.g., medical training)

21.11.201729

Social media

• Snapshat• Cinco de Mayo lens

• Facebook• Camera masks

• Not real time (yet)

21.11.201730

X-RAY Visualisation

21.11.201731

https://www.youtube.com/watch?v=OcUTlEMTUsUhttp://www.inautonews.com/x-ray-vision-gives-volkswagen-a-virtual-training-advantage

Maintenance

21.11.201732

https://www.youtube.com/watch?v=Y5ywMb6SeGc

Interior design

21.11.201733

https://www.youtube.com/watch?v=wITqCim0nTM

Advertising

21.11.201734

https://www.youtube.com/user/layarmobile

Possible Future applications

Social interactionhttps://www.youtube.com/watch?v=tb0pMeg1UN0

AR Sci-Fi video, Sight:https://www.youtube.com/watch?v=lK_cdkpazjI

AR-glasses concept videos:https://www.youtube.com/watch?v=FM-Cqr4mYoA

21.11.201735

From Real to Virtual

21.11.201736

[1] Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, E77-D(12), 1321–1329.

Augmented Virtuality

21.11.201737

https://www.leapmotion.com/product/vr

http://basilic.informatik.uni-hamburg.de/Publications/2009/SBRH09a

Virtual Reality

The environment is purely virtual

Two approaches1. Cave automated virtual environment

(CAVE) 2. Head-mounted display

(HMD)

21.11.201738

Kuva 6: Eri versiot Oculus Rift -laseista. Ylhaalla kehittajien version 1 ja 2, alhaalla

ensimmainen kuluttajaversio.

Ominaisuusvertailu on Taulukossa 1. Ominaisuuksista huolimatta tata versiota myytiin

65,000 kappaletta. (Ostrow et al., 2015)

Oculus Rift

DK1 DK2 CV1 Silma

Nayton resoluutio / silma 640x800 960x1080 1080x1200 17000x13500

Virkistystaajuus 60 Hz 75 Hz 90 Hz -

Nakokentta 110 astetta 100 astetta 110 astetta 270 astetta

Yhden silman 109�H, 113�V 94�H, 106�V 80�H, 90�V 170�H, 135�V

Kahden silman 117�H, 113�V 95�H, 106�V 88�H, 90�V 200�H, 135�V

Paino 380 g 440 g 470 g -

Taulukko 1: Oculus Rift myyntiversioiden vertailu. (Desai et al., 2014)

17

Key Characteristics of Virtual Reality

• Fast and accurate tracking• More controlled environment than in AR

• 3D Stereoscopic display• Wide field-of-view display

21.11.201739

Virtual reality experience

ImmersionPresence• Feeling of

“being there”

21.11.201740

https://www.youtube.com/watch?v=pAC5SeNH8jw

Possibilities for VR

Games

Simulators

Remote collaboration

360 Video recording

21.11.201741

Challenges in VR experience

• Visual discomfort• Accommodation and convergence

mismatch, wearing display • Motion sickness

• Mismatch between visual and vestibular systems

• Limited resolution • Oculus Rift, Consumer Version

1080x1200, Human eye about ten times more

21.11.201742

Jason Geng, "Three-dimensional display technologies," Adv. Opt. Photon. 5, 456-535 (2013)

Challenges in VR experience

How to implement moving in virtual space?

21.11.201743

Challenges in VR experience

• Content creation• Virtual content creation

• 3D software (e.g., Unity3D)

• Content from real environment• 360 Video recording

21.11.201744

21.11.201745

AR

VR

Next week’s lecture

Prof. Antti Oulasvirta will talk about computational user interfaces

Note: Location is AS1!

21.11.201746

Conclusion

Do not believe in hype, there is still a large gap between real and virtual environment

In mixed reality the key is a meaningful interplay between physical environment and virtual environment

21.11.201747

Further reading

Course book (Benyon’s) Chapter 13: pp. 291 – 293

Furht, B. Handbook of Augmented Reality, 2011, Springer

Mark Billinghurst’s lecture notes in SlideShare: http://www.slideshare.net/search/slideshow?searchfrom=header&q=COMP+4010&ud=any&ft=all&lang=**&sort=

21.11.201748