Mobile Augmented

Reality

Mobile Augmented

Reality

Dieter SchmalstiegDieter SchmalstiegGraz University of Technology, Graz University of Technology,

AustriaAustria

Mobile Augmented RealityLecture Overview

Mobile Augmented RealityLecture Overview

• Motivation

• Hardware requirementsHardware requirements

• TrackingTracking

• Environmental modelingEnvironmental modeling

Mobile AR – MotivationMobile AR – Motivation

Mobile, wearable computing opens Mobile, wearable computing opens up new possibilities up new possibilities

• location-aware/situated computing

Now, the interface is everywhere Now, the interface is everywhere

•AR is a powerful UI for this type of computing

Mobile AR – MotivationMobile AR – Motivation

Mobile AR Mobile AR Applications:Applications:

• Navigational aids

• Communication aids

• Personal situated information DB

• General UI for appliances

• Tourism

• Journalism

• Maintenance and construction

• Military training and warfighting

Mobile AR – BackgroundMobile AR – Background

Post-WIMP interfaces:Post-WIMP interfaces:

3DDesktop 3D, Desktop VR, Fishtank VR

Projection-based VR

Head-mounted VR

SituatedMobile, Wearable, Handheld

Multi-Device, Pervasive

Tangible, Embodied

MultimodalSpeech, Gestures, Audio, Haptic

UbiquitousComputing

Mobile AR

Mobile AR – BackgroundSteps Toward Wearable ComputingMobile AR – BackgroundSteps Toward Wearable Computing

ComputerComputerForm Form

FactorFactor

RoomRoomWallWallDeskDeskBox Box Laptop Laptop Palmtop Palmtop ClothingClothing

UserUserRelationshipRelationship

SubmitSubmitShareShareSit atSit at

. . . and carry . . . and carry before/afterbefore/after

HoldHoldWearWear

Implications of Wearability(after S. Mann, B. Rhodes, T. Starner)

Implications of Wearability(after S. Mann, B. Rhodes, T. Starner)

MobilityMobility• usable/used indoors and outdoors

IntimacyIntimacy• sense the wearer’s body, communicate privately

Context sensitivityContext sensitivity• take into account changing environment

ConstancyConstancy• Permeation of UI into wearer’s life

Mobile AR – BackgroundSituated ComputingMobile AR – BackgroundSituated Computing

Ubiquitous computing (Weiser ’89)Ubiquitous computing (Weiser ’89)

PARCTab (1993)PARCTab (1993)

Hull et al. (’97) state that Hull et al. (’97) state that “ “situated computing concerns the ability situated computing concerns the ability

of computing devices to detect, of computing devices to detect, interpret, and respond to aspects of the interpret, and respond to aspects of the user’s local environment”user’s local environment”

Mobile AR – BackgroundWorldBoardMobile AR – BackgroundWorldBoard

1990s: many researchers 1990s: many researchers started to co-locate started to co-locate information with physical information with physical space space

J. Spohrer 1996: What comes J. Spohrer 1996: What comes after the World Wide Web?after the World Wide Web?

Information in place. Information in place. The world as a repository of The world as a repository of information. (Imagined as a information. (Imagined as a service Apple Computer, service Apple Computer, Inc. would provide.)Inc. would provide.)

What is Mobile AR?Ways of augmenting a mobile user’s environment

What is Mobile AR?Ways of augmenting a mobile user’s environment

• wearable display, no tracking whatsoever

• body-stabilized wearable display (orientation tracking only)

• location-dependent audio augmentation (with or without spatialized audio)

• location-dependent screen-stabilized augmentation (possibly monocular)

• location-dependent body-stabilized augmentation(on a projection cylinder/sphere surrounding the user)

• stereo head-tracked, position tracked, AR with full overlay registration

Ideal Mobile AR DeviceIdeal Mobile AR Device

+ A pair of stylish + A pair of stylish sunglassessunglasses

+ Hi-res stereo 3D + Hi-res stereo 3D graphicsgraphics

+ Built-in computer + Built-in computer with wireless networkwith wireless network

+ Highly accurate + Highly accurate 6DOF tracking6DOF tracking

+ All for $99.90+ All for $99.90

- Doesn‘t exist (yet?)- Doesn‘t exist (yet?)

Challenges of Mobile ARChallenges of Mobile AR

Challenges of mobile computingChallenges of mobile computing

• Limited resources

• Size, weight

• Battery live

• Ruggedness

Challenges of mobile ARChallenges of mobile AR

• Tracking, 3D graphics, real-time performance

Challenges of outdoor environment Challenges of outdoor environment • Lighting conditions

• no instrumented environment possible

Components of Mobile ARComponents of Mobile AR

Overview:Overview:

Computing platformsComputing platforms

DisplaysDisplays

TrackingTracking

Computing platformsComputing platforms

Mobile computerMobile computer

form factorsform factors

NotebookNotebook

Tablet PCTablet PC

Wearable PCWearable PC

PDAPDA

SmartphoneSmartphone

AR Backpack ExamplesAR Backpack Examples

Columbia Touring Machine(2002 )

Rockwell vest (1999)

AT&T Sentient AR(2001)

1997

Display platformsDisplay platformsMinolta Forgettable Display

MicroVision Nomad retinal scanning display

MicroOptical EG 7

Sony GlasstronStereo optical see-thru

Head-mounted displayHead-mounted display•Monocular, monoscopic, stereoscopic

•Optical/video see-thru, see-around

Handheld display (PDA etc.)Handheld display (PDA etc.)

Audio displayAudio display

TrackingTracking

RequirementsRequirements•Provide position and orientation

(separately?)

•Untethered, large working volume

Indoor vs outdoorIndoor vs outdoor• Indoor: Can instrument environment

•Outdoor: Self-contained or satellite based

Indoor TrackingIndoor Tracking

Ultrasonic beacon arrayUltrasonic beacon array• AT&T Bat, Intersense IS900

Infrared LED arrayInfrared LED array• UNC HiBall, MIT‘s locust swarm

Outside-in computer visionOutside-in computer vision• Observer cameras + passive IR targets

(e.g., ARTTrack - medium range)

Inside-out computer visionInside-out computer vision• Fiducials (e.g. ARToolKit at Uni.SA)

• Instrument environment with fiducials

Dead-reckoning techniquesDead-reckoning techniques

Outdoor trackingOutdoor tracking

GPS, enhanced GPSGPS, enhanced GPS• differential GPS

• RTK

• Pseudolites

• Dead-reckoning

Inside-out computer visionInside-out computer vision• Natural features, e.g. USC, TU-Graz

• Still challenges in performance + robustness

OrientationOrientation• Gyroscopes, magnetometers, inclinometers

Some Pioneers of Outdoor ARSome Pioneers of Outdoor AR

Columbia University: MARS Columbia University: MARS ‘97‘97

Uni.SA: TinmithUni.SA: Tinmith

Naval Research Lab: BARS

Receiver chain

VGA

Serial port

qinertial = [(qx, qy, qz), qw]

Inertialtracker

Receivers

Sentient Computing

Server

Case Study:AT&T Cambridge’s Sentient ARCase Study:AT&T Cambridge’s Sentient AR

Memory prodMemory prod

MaintainenceMaintainence

Everywhere GUI

Case Study:Studierstube Mobile AR System

Case Study:Studierstube Mobile AR System

HMDDGPS

Inertialsensor

camera

notebookWLANGPRS modem

trackedtouchpad

Wide Area Tracking- DGPS (outdoors)- ARToolKit (indoors)

Indoor + outdoor

Navigation & BrowsingNavigation & Browsing

indoorsindoors outdoorsoutdoors

Ergonomic ConsiderationsErgonomic Considerations

How to make wearable AR ergonomically andsocially acceptable?

Surprising AnswerSurprising Answer

We can start today!We can start today!

„„Handheld Handheld Augmented Reality“Augmented Reality“

Platform: PocketPCPlatform: PocketPC

Fully self contained visionFully self contained visiontracking + 3D graphicstracking + 3D graphics

Environmental ModelingEnvironmental Modeling

Unless we are attaching information to markers in the scene only,the computer needs a model of the environment

• For annotating detailed infrastructure: need geometrical model

• Access to DB of environmental information

Environmental ModelingEnvironmental Modeling

• Model urban infrastructure from 2D topographic maps and aerial photographs

• Modeling from laser range finder data

• Modeling from a combination of a set of photographs and geometrical constraints (Berkeley Façade, Canoma)