Augmented and Virtual Reality - Massachusetts Institute of...

of 51 /51
Augmented and Virtual Reality 6.S063 Engineering Interaction Technologies Prof. Stefanie Mueller | MIT CSAIL | HCI Engineering Group

Transcript of Augmented and Virtual Reality - Massachusetts Institute of...

Page 1: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

Augmented and Virtual Reality6.S063 Engineering Interaction TechnologiesProf. Stefanie Mueller | MIT CSAIL | HCI Engineering Group

Page 2: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

AugmentedReality(AR)

mixed reality

real environment augmented reality (AR) augmented virtuality (AV) virtual reality (VR)

AR supplements the real world VR replaces the real world

Page 3: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

let’s look at thevery beginning…

Page 4: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

not quite the first…

when was the first AR/VR headset developed?

<30 second brainstorming>

2012 - Occulus Rift

Page 5: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

1965: Ivan Sutherland - ”Sword of Damocles” (first video headset)

Page 6: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

binocular display (both eyes) displayed output

1965: Ivan Sutherland - ”Sword of Damocles” (first video headset)

Page 7: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

• too heave to carry, thus suspended from ceiling

• designed for helicopter pilots to have light synchronized with head movement when flying in the dark

• included head & eye tracking

1965: Ivan Sutherland - ”Sword of Damocles” (first video headset)

Page 8: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

a room where the computer can control the existence of matter

‘the ultimate display’::

(Ivan Sutherland, 1965)

Page 9: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

Ivan created the first head mounted display…

but the idea of augmenting human senses has already around a bit longer…

Page 10: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

1965: Marc Heiling - Sensorama (even before Sutherland)

• stereoscopic display • vibrating seat • body tilting • fragrance generating system • wind generation

Page 11: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

1985: NASA virtual environment work station

firstdata glove

Page 12: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

1985: NASA virtual environment work station

Page 13: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

1985: NASA virtual environment work station

Page 14: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

1985: NASA virtual environment work station

Page 15: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

any ideas which tech advances made AR/VR suddenly mass-available in the last 5-10 years?

<30 second brainstorming>

Page 16: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

• small and light-weight displays became low-cost• mobile computing: small & powerful computers for graphics

Page 17: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

1996: Columbia - Mobile AR ”Touring Machine:”

firstmobilesystem

Page 18: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

2000+: consumer AR / VR

commodity devices are finally sufficiently powerful, small, inexpensive

Page 19: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

what’s next in terms of technology? how does the next AR/VR hardware look like?

<30 second brainstorming>

Page 20: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

AR contact lenses (many challenges, max. 8 pixel right now)

Page 21: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

display technologies

Page 22: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

Head%attached))))))))))Body%attached)))))))))))))))))))))Spatial

hand-helddisplay

spatial see-through

display

head-worn

display

retinal display

projector

projector

[Ramesh Raskar]

Page 23: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

retinal displayHead%attached))))))))))Body%attached)))))))))))))))))))))Spatial

hand-helddisplay

spatial see-through

display

head-worn

display

retinal display

projector

projector

Page 24: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

[NaturalVue, ColorMax]

future• AR with digital overlay • future: VR

today• multifocal lenses • color blindness lenses • sensor lenses for body information

Page 25: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

head-worn displaysHead%attached))))))))))Body%attached)))))))))))))))))))))Spatial

hand-helddisplay

spatial see-through

display

head-worn

display

retinal display

projector

projector

Page 26: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

Vuzix Star 1200XL

optical see-through:• you see the real-world with your eyes in

the same way as using regular glasses • on top digital content is displayed

Page 27: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

using a cellphone and materials from the workshop, how would you build an optical see-through display?

draw it on paper.

<2 minutes>

Page 28: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

smart phone display

real world

semi-transparent mirror

optical see-through:

Page 29: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

optical see-through:

display

mirror

Page 30: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

what are some of the benefits and drawbacks of optical see through?

<30 second brainstorming>

Page 31: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

• real world is full resolution and FOV• if you run out of battery, you still see something

benefits::

• no control over real world • you cannot display sth behind a real world object• virtual/real worlds focus at different distances

Page 32: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

Vuzix Wrap 1200AR

video see-through:• real world is captured with cameras • camera image is digitally enhanced • you look at the camera image

Page 33: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

full control over image you can display something behind a real world object

real world is low resolution and low field of view (FOV)

benefits::

Page 34: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

field of view (FOV): here HoloLens, FOV < 100 degree)

Page 35: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

hand-held displaysHead%attached))))))))))Body%attached)))))))))))))))))))))Spatial

hand-helddisplay

spatial see-through

display

head-worn

display

retinal display

projector

projector

Page 36: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

kind of video see-throughdevice camera captures real world, then displays it on screen

Page 37: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

markers help to align multiple devices

Page 38: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

spatial augmented reality (SAR)Head%attached))))))))))Body%attached)))))))))))))))))))))Spatial

hand-helddisplay

spatial see-through

display

head-worn

display

retinal display

projector

projector

Page 39: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

spatial AR:• project onto real-world objects • no need to wear or hold something • digital content directly in the real-world

Page 40: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

spatial AR:• camera: find out where the object is • projector: enhance with digital content

[SoftAR, Punpongsanon]

Page 41: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

Head%attached))))))))))Body%attached)))))))))))))))))))))Spatial

hand-helddisplay

spatial see-through

display

head-worn

display

retinal display

projector

projector

?[Ramesh Raskar]

Page 42: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

if spatial AR brings digital content closer to the real-world object, what would be the next logical step in terms of technology?

<30 second brainstorming>

Page 43: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

everything becomes a ‘display’the future?

Page 44: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

“The ultimate display would, of course, be a room within which the computer can control the existence of matter.

A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal.”

— I. Sutherland, The Ultimate Display, Proc. IFIP 65, 506–508, 1965

and maybe even a ‘physical display’…

Page 45: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

any many moreinteresting questions..

Page 46: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

Liwei Chan: Frontface

Page 47: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

Yoichi Ochiai: Augmented Mom

Page 48: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented
Page 49: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

that’s it for today…

Page 50: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

until wednesday• laser cut business card (HW1) • setup your micro-controller on your computer (HW2) • connect your micro-controller to wifi (HW3)

Jared’s tuesday office hour: 4.15 - 6pm

Page 51: Augmented and Virtual Reality - Massachusetts Institute of ...groups.csail.mit.edu/.../2017-fall/5-augmented-reality.pdf · Augmented Reality (AR) mixed reality real environment augmented

end.