Ubiquitous Computing - DFKIkrueger/teaching/Website/Ubicomp_2009_2010_files/Ubicomp... · Weiser:...

October 14, 09 Saarland University Ubicomp A. Krüger 1

Ubiquitous Computing

Winter term 2009

Prof. Dr. Antonio Krüger

10 December 2009 Ubicomp Antonio Krüger Winter term 20092

Topics Today

Context Awareness• Definitions• Example projects• The Context Toolkit• Low level context acquisition• Some implementation concepts


Trying to define context

! Context acc. to Bill Schilit (Active badge 1992 + Parctab 1994)• „mobile distributed Computing“: Mobile

Computers and Users• The three (spatial) Ws:

- Where is the user?- Who is near the user?- What services can the user use in the vicinity?

• Important: the temporal changes of the 3 Ws

10 December 2009 IfGi Universität Münster Ubicomp A. Krüger4

Trying to define context (2)

! Context properties• Location• Lighting/brightness• Sound/noise• Network availability• Network bandwidth• Group constellation

• Dialogue history• Position history• User interest• Time of day• Orientation • Speed • Temperature


Example project: Active BadgesOlivetti / AT&T, Schilit, Hopper, Harter, et al.

! Teleport• Redirect screen output

from "home" computer to nearby computer

! Phone forwarding• Automatically forward

phone calls to nearest phone


Active Badges: Technology! Badges emit infrared (IR remote)

signals! 1 signal every 15 sec.! Avoid 2 badges in sync

• use high tolerance components• Light sensor changes interval• " switched off when in the dark

! Button to trigger events! Sensors distributed in the building! Central server scans regularly for

„badge sightings“! Over 1500 badges and 2000

sensors used worldwide

“A disadvantage of an infrequent signal from the badge is that the location of a badge is only known,at best, to a 15-second time window. However, because in general a person tends to move relativelyslowly in an office building, the information the Active Badge system provides is very accurate.” ;-))


Active Badges: Initial Services! FIND (name)

• Provides the current location of the named badge and, if it has recently moved, a list of all the locations it has been sighted at in the last five minutes along with the likelihood of finding it at each.

! WITH (name)• Locates a named badge and provides information about other badges that are in

the immediate locality of that badge.! LOOK (location)

• Allows an investigation to be made of the badges that are currently near the specified location.

! NOTIFY (name)• An alarm mechanism that generates an audible indication of when the named

badge is next sighted after executing the command. ‘NOTIFY’ is particularly useful when trying to deliver an urgent message to a member of staff who is out of the office on business for long periods of time.

! HISTORY (name)• Generates a condensed report of the location history for the named badge during

a one-hour period. The system intentionally does not record any location data on a permanent storage medium, to dispel concern about long-term monitoring of an employee’s movements.


Active Badge Video (Hopper et al CHI 93)

8


Xerox ParcTabhttp://sandbox.parc.xerox.com/parctab/

! Infrared network• Base stations in the ceiling• Low bandwith, modulated carrier• Transmission radius ~7m

! Mobile tab-sized devices• Unistroke input via pen

! Context-aware applications:• Information access• Communication• Collaboration


Information Access

! Weather (Internet/local)! Dictionary, Thesaurus ! UNIX file browser ! WWW browser (mit Einschränkungen)! Calendar manager (Sun's cm) ! Dateimanager (ortsabhängig)


Communication

! email: permanent access! pager! locator! „Communicator“, media-space controller

• Tab proposes best communication devices in the surroundings and initiates connection


Collaboration

! Tab as pointing device • Remote pointer control for liveboard• Move pointer with pen on the tab screen

! Tabdraw: collaborative drawing• One canvas per room

! Arbitron (Tool for voting)• Tell the presenter to speed up or slow down


Other Applications

! Remote Control• Control of physical environment• Universal (self-configuring) remote control

! Local (on Tab) applications• For offline use• Note pad for memos


Techniques in Context-Aware Computing (Schilit et al. 1994)

! Proximate selection• E.g., list closes devices first

! Automatic contextual reconfiguration• E.g., automatically select

nearby devices for output! Contextual information &

commands• Commands with different

meanings in different contexts! Context-triggered actions


Context-triggered actions! Simple If-Then Rules, similar to Unix CRON-

Demon:• Coffee Kitchen arriving „play -v 50 /sounds/rooster.au• schilit * attention „emacs -display $NEARESTHOST:0.0“

! Contextual reminders: information is displayed under certain conditions. Example:

$DATE=„after April 15“ AND $TIME=„after 10“ AND $room=„35-2-200“ AND $WITH-USER=Adams“AND Color($DISPLAY)=„true“


Context-sensitive Systems

! Scientific problems• How to recognize relevant context (Sensors)• How to use the obtained context information

(adaptive/reactive Systems)- Implicit vs. Explicit control of systems- Reactive Systems- Situated Systems- Adaptive Systems


Situated Systems

! Activity as context• Location and identity

- Simple, but only weak characterization of situations• Complex Sensors (e.g., cameras)

- rich information- high computing power needed- endangered privacy

• Integration of many simple sensors- TEA Project Karlsruhe - Context Toolkit


TEA ProjectEU-funded 1998-2000, TECO, Starlab/Be, Nokia/Fin

! TEA: Technologies for Enabling Awareness• Combination of simple sensors instead of

complex image processing• Combine multiple sensors with context

information • HW/SW-Addon for mobile devices

- low-energy, low-cost- Target platform: cellular phone


TEA Project! GSM Telephone as the TEA Host

• User expects different behavior of the phone according to the situation

• State of the art: manual profile selection• TEA enables automated profile selection

depending on sensor data! Application Context-Call

• Caller dials the number of the user, and is told the user’s situation by TEA (e.g., „In a meeting“)


TEA Project! TEA-Hardware:

• light, audio, acceleration, and temperature sensors• Microcontroller controls the sensors and extracts hints• Final decision is made on the host

First prototype second Prototype


Active artifacts

! Concept:• Determine activity where it occurs• Add “self perception” to everyday things• Communicate their own state• The artifact digitally “supports” its own

applications• Example: MediaCup

- http://mediacup.teco.edu/


Mediacup (Teco, Univ. Karlsruhe)

! First experimental „active artifact“! Technical Info:

• PIC-Microcontroller, 15k/384Byte, low-energy• IrDA physical level communication• 3 acceleration, 1 weight, 1 temperature sensor• 2 condensers as power supply

! „Self perception“:• Reading out sensors periodically• Compute important events : in the shelf, full, empty,

currently in use, etc...


Mediacup! Small number of cups

! In use since 1999

! 95% correct recognition of Multi sensor events

! Important design criterion: Energy consumption, heavy influence on outward appearance


Context acquisition through Cooperation! Cooperating Arteficts

(Strohbach et al 2004)

! Knowledge is exchanged between artefacts that sense their local context

! Example: chemical containers


Each Artifact is context sensitive


Sharing knowledge between artifacts


Fact base


Inference rules


Hazard Examples

Hazard critical mass!Hazard incompatible


Limited Resources

! Technical resources (of the environment)• Available media: e.g. displays, loudspeakers• Media attributes: screen size & resolution,

colors• Quality of positional data: user’s location (e.g.

indoor/outdoor), orientation and speed• Available CPU-power and memory• Communication bandwidth


Limited Resources (2)

! Cognitive Resources (of the user)• Cognitive load:

- Use of working memory- Time pressure

• Familiarity with the environment• Personal preferences:

- Media, content and presentation styles• Limited vision, hearing, motor skills, etc.• Communication abilities: limited use of

modalities, e.g. use of gesture and speech


Types of resource adaptivity

! Adapt to available resources• Technical resources• Cognitive resources

! Resource-adapted Systems• Systems which are optimized towards a certain

resource limitation• Result is optimized to the limitations of the sensors in

the environment.• Different resource situations lead to failure• Examples: TEA, MediaCup


Types of resource adaptivity

! Resource-adaptive systems• Implementation of a strategy to adapt to a limited

resource• Result is improved with increasing availability of

resource (example: any-time algorithm)

! Resource-adapting systems• A Meta-strategy chooses between different resource-

adaptive strategies to fit the resource limitations• Implementation of a meta-cognitive system level


Context Toolkit (with slides courtesy of Anind Dey)

! Anind K. Dey (CMU Pittsburgh)! Toolkit to support Context-Aware

applications! Strong formalization of “context”! Implementation in Java. ! Can be distributed on several machines

in the environment


Context and Context-Awareness

! Focused on input

! Context: any information relevant to an interaction that can be used to characterize the situation of an entity

! Context-Awareness • General model of interactive computing• Addresses subset of ubicomp problems: input


Value of Context

! Potential for improved usability• Very important for mobile users with poor input

devices

! “Smarter” applications


Example

! Tour guides, travel assistants, personalization software

! Reminder to buy milk• When to deliver: not time/location specific• How to deliver: appropriate modality


Building Applications

! M. Weiser: The whole point of ubiquitous computing, of course, is the applications.

• But … what if the applications are hard to build? And, what if this inhibits our ability to build compelling applications?


Why Context is Hard to Use

! Acquired from sensors• Not just keyboards and mice – lots of heterogeneous

devices

! Need to abstract data

! Distributed

! Dynamic


Results of Difficulties

! Ad hoc application building• Difficult to build, reuse and evolve

! Small variety of sensors! Small variety of context: mostly location ! Few applications, mostly simple: mostly

presenting context

! Practical: difficult to prototype, test and evaluate


Toolkit Requirements

! Context specification• Discovery

• Storage• Separation of concerns

• Interpretation• Transparent communications• Constant availability


Look to input handling ! Graphical User Interface (GUI) widgets

• separation of concerns• callbacks and attributes• query/subscribe• common interface

• e.g. button


Context Widgets

! Responsible for acquiring and abstracting data from particular sensor, separation of concerns, storage

Widget

Sensor

Widget

Application Application

Sensor

ContextArchitecture Face

Recognition

Location Location

In/Out Board

Smart Card Reader


Context Interpreters

! Convert or interpret context to higher level information

! Context not available at appropriate level

Face Recognition

Location Location

In/Out Board

Smart Card Reader

ID to NameInterpreter


Context Aggregators

! Collect context relevant to particular entities (recall definition)

! Further separation, simplifies design

Face Recognition

Location Location

In/Out Board

Smart Card Reader


BuildingAggregator


Context Toolkit Framework! Supports real-world model/methodology and provides library

(distributed: XML/HTTP, input-focused)! Component model: facilitates building of applications

Widget

Sensor

Widget

Application Application

Interpreter InterpreterAggregator

Sensor

ContextArchitecture

Service

Actuator


Experiences: Benefits

! Provides separation of concerns

! Lightweight integration and re-use of components

! Easy to create and evolve apps, allowing exploration of the design space• Add context to context-less apps• Add more context to context-aware apps


Aware Home (MANSE ’99)

! Great testbed for context-aware computing! 3 goals: elderly, infants, everyone! Context Toolkit is the s/w infrastructure in the

Aware Home


Applications Built

! Simple use of location:• Turn lights on and off (perform service)

! Location and id (perform service)• Information Guide: present info about user’s

group (CHI ’99)

• Context-Aware Mailing List


In/Out Board – 3 versions (CHI ’99)

! Context used: location, identity, time! How used: present context


In/Out Board Architecture

LocationWidget

In/Out Board

iButtonDock


Web-basedIn/Out Board

Context-AwareMailing

List

PinPoint3D-iD

• Simple app demonstrates support for reusability (don’t have to re-build infrastructure on per-application basis) and evolving applications


Serendipitous Meetings• Context used: location, id, time, activity • How used: present, perform service, tag• record and tag drawings and audio for later retrieval


Playbackcontrols

Filters

Ink written beforecurrent time is in original color

Ink written aftercurrent time is in lighter color

Current time within sessionSelected session

Selected dayDay containing whiteboard activity


Meeting Architecture

ContextArchitecture

For each possible location of the

mobile board

LocationWidget

DUMMBO

LocationWidget


iButtonDock

iButtonDock


Conference Assistant (ISWC ’99)

! Context used: location, multiple levels of identity, activity, time

! How used: present, service, tag

Slide

User Notes

Interest Control

Audio/VideoIndicator

Slide text User notes

Retrievedslide

Query Interface

Schedule

context widgetsIdentity, Location, Activity

Joe Smith context


Conference Assistant Arch.


Low level context recognition(Cakmakci et al. 2002)

! Design “context aware hardware”

! Enhance wearable computing

! Detect simple user activities, like sitting, walking, looking at the watch


Low level context recognition! Use statistical modeling techniques from robotics

to determine context! Use accelerometers to record movement changes! Apply Bayes rule to determine probability of certain

contexts:


Low level context recognition! First simple Experiment: Detect whether users

are sitting, standing and walking

Acceleration over 4500 data points during the experimentUse the first 1000 data points for learning


Low level context recognition

! Results of recognition



! 2nd experiment: Detect when users glance at their watch

! Idea: reduce power of watch (e.g. toggle display) during use

! Extremely important for a wristwatch computer (IBM WWC: ARM7 processor, 8Mb flash memory, 8Mb of DRAM, serial, IRDA, and expansion interfaces).



Evaluation set in a blind experiment



Use a single hypothesis approach to model “wrist watching”(Only one well defined class)



Results:Probability of certain data points belonging to a wrist watch action


Intelligent adaptive assistants

indoor

outdoor

Two main objectives:

• Generate adapted route descriptions• Generate adapted presentations

Museum

Exhibit

Virtual Window

PDA

REAL:Resourceadaptive Navigation

PEACH: Personal Experiences in Active Cultural Heritage


Research Problems

! Given an instrumented environment and a user’s task• How to adapt route descriptions to

- the technical resources of the environment?- the cognitive resources of the user?

• How to adapt presentations to - technical resources of the environment?- user preferences and interests?


Approach

technical resources X cognitive resources X user’s task

?

Reaction of the intelligent environment• Content presentation• Interaction possibilities

A2: Use a declarative approach to specify system behavior under limited resources, represent what has to be presented!

A1: Design an architecture that supports resource adaptivity at several levels.


SpatialKnowledge

RouteKnowledge

SurveyKnowledge

LandmarkKnowledge

Segment

Task

SegmentTask

Directions

ObjectDescription

AdditionalInformation

SegmentPresentation

SupportTechniqueTechnique

Abstraction

SketchGeneration

MetaGraphics

PresentationAgent

CameraTechnique Shot

Zoom

Pan

Movement

World InMiniature

AnimatedWalkthrough

MultipleViews

GraphicalPresentation

ResourceRestriction

CognitiveRestriction

PositionResolution

OrientationResolution

TechnicalRestriction

DeviceRestriction

GraphicCapabilities

Bandwidth

CognitiveLoad

TimeConstraints

Comm.Abilities

WorkingMemory

SpatialFamiliarity

SystemKnowledge

DesignKnowledge

DomainOntology

UserModel

3DModel


Adaptive graphical route descriptions

Resource sensitiveroute

Wayfinding

Presentation script

Presentation PlanningLimited Resources- technical- cognitive

Media specificInformation

Presentation

Adaptive routedescription


Hier weiter

71


Resource-adaptive route finding

Optimization criteria:

• Shortest (fastest) route • Reduce amount of turning points, • Reduce complexity of turning points• Exploit spatial familiarity

A

B


Planning Formalism

Show-way-description

sketch-draw-arrow

sketch-show-start-point

sketch-draw-circle sketch-draw-cross

sketch-show-end-pointsketch-show-trajectory

sketch-show-trajectory-step

sketch-show-trajectory-segment sketch-show-decision-point

additional

show-sketch show-3d-walkthrough

alternative

sketch-show-path sketch-show-ground-plan

incremental

sketch-draw-thumbnail sketch-write-label

conditional


Representing the domain ontology


A Hybrid Navigation System

Presentation

Server

Information Booth

Mobile Indoor

Mobile in/outdoor


Information Kiosk: Techniques


Camera Styles

Egoview

Front and shoulder camera

Birdʻseye view

• Position

• Rotation• Viewing angle


Information Kiosk:Presentation Agent


Graphical Techniques and Types

bad -- - 0 + ++ good

Our approach: hybrid representation of spatial knowledge • Geographic coordinates

– Geometrically precise– Data acquisition– Visualization (mainly for outdoor purposes)– Example: WGS84 coordinates

• Symbolic location identifier– Geometrically abstract/vague– Used to describe situations– Visualization (mainly indoors)– Example: model of a spatial ontology

Related Work• Hybrid Models [Leonhardt 98]

– Combines symbolic and geometrical location descr.• Hierarchical Location Identifier [Jiang02]

– Combines symbolic and geometrical location descr. in a single identifier

• Nexus [Dürr03]– Symbolic object-graph for geo-broadcasts

• Problem: location description is not embeded into the situational context of the user:– No relation to user prefrences and goals– Reduced adaptation of the interaction

User Modeling and Spatial representations (Heckmann 2005)

UbisWorld ( Ontology + Instances )

Physical Ontology

Spatial Ontology

Temporal Ontology

User/Group

Activity Ontology

Classes & Predicates

Device/Object Location

Individuals (id, label, category, parents)

Time 3 47

2 13 4 5

Relations

Situation Ontology

28

OWLDamlOILRDFSQL

Levels of description Instance of concept “room”

Building 36

Info !

Campus

Info !

Building 45

Info !

Building 28

Info !

Room 124

Info !

Hallway

Info !

Kitchen

Info !

2nd Floor

Info !

URI=//www.u2m.org/#b36

URI=//www.cs.uni-sb.de/#b36Room124

Geometric_model

Geometric_model

Symbolic-model

Symbolic-model

Refinement

Parent

Spatial Granularity Level: Street - Building

Map of University Campus

Map of 2nd Floor

Spatial Granularity Level: Floor - Room

URI=//www.uni-sb.de/#CampusMap

URI=//www.cs.uni-sb.de/#b362ndFloor

N

0

Infrared beacons

RFID-beacons

Mapeditor Yamamoto

Permeable edges

Selected polygon

Polygons representing rooms

Symbolic identifier

Mobile Multimodal Interaction (M3I) Plattform• Improved interaction with small screen devices

– Navigation instructions and spatial exploration• Mobile 2D- und 3D-Graphics • Embedded speech recogntion and synthesis

– Commands, Route information, Landmarks• Gesture recognition• Embedded into an infrastructure

– Positioning– Routing– Local maps

• Universal platform– Special target groups (e.g. visually impaired users)– Enables Empirical evaluations in the field

Implementations-plattform:Pocket PC

Mobile Graphics

2D-Graphics 3D-Graphics Ego-perspectiveTop-down views

Mobile gesture recognitionTypes• Intra – gesture on mobile screen• Extra – gesture in the world

Intra-gesture library• points (buildings)• lines (streets)• circles (areas)

Sensors:

Intra gestures: Point and slide Extra gesture

Magn.

Speech recognition and synthesis

„Command and Control“ Landmarks, point gesture Street, line gesture

Fusion with Intra-gesture

Example M3I

Positioning technology in M3I• Local sensor-data collection and remote

processing– RFID– infrared– GPS– Self localization (Kray 2004)

• Determining the position – Mobil

• Advantage: no infrastructure needed, robust– infrastrucutre

• Advantage: use of external services possible

IRRFIDRFID-Leser

Indoor positioning

RFID-beacon

Infrared-beacon

Strong Signal Str.

Weak Signal Str.

Route

Actual position

Receive ID of IR-beacon

A hybrid method for indoor localisation

Activate RFID-beacons in the vicinity

9


1035

1036

1034

Receive responses of RFID-tags (ID and signal strength)

1039


Spatial Audio (2 von 8)

Plasma-screen(touch sensitive)

RFID shelf

Projector-Camera Unit

Products with RFID-tags

IR Beacons

Optical Markers

REAL: Instrumented Environment S.U.P.I.E.

REAL architecture: Navigation

Communication using Blackboard Infrastructure

PocketPC

GISGeometric Location

Model

Local Mapand Route

UbisWorld• Locations• Persons• Objects• Activities

IRRF

GPS

PedestrianNavigationAssistant

Shopping Cart

RF Antenna

TabletPC

SpeechStylus

Shop AssistantPositioning bySensor Fusion

Location

ID

Routing

PresentationPlanner

SpeechStylus

Speech + Gesture

Product ID

Presentation

Plasma Display

RFID Shelf

RF Antenna

Request

Smart Door Displays

Infrared BeaconRF Beacon

Productwith RFID Tag

ID CardCybercode

ID CardRFID Tag

Smart DoorDisplays

Beamover

Touchscreen

Spatial Audio

Search Light

Pedestrian Navigation Assistant

Presentation

PlansKno

wle

dge

Personal Public

Ser

vice

Sen

sing

Use

r Int

erac

tion

Pos

ition

ing

M3I Platform Assistance through instrumented space

Navigation in instrumented spaces

Smart Door DisplaysMixed reality (projection)Mobile client

Indoor Navigation with Smart Door Displays

Example

Multi-user navigation w. public displays(ruzkio,schmidt,krüger, CHI 2005)

• Synchronized signals on personal and public devices

• Vibration alarm of mobile phones as a specialization of a one-bit display

Initial User study

• Lab setting with 14 participants• “Rotating compass” projected on the floor• In 140/140 trials participants identified the right direction• Two modes: “phone in hand”, “phone in pocket”• Problem: What if more than one display is in sight?• Problem: What if the phone is carried in a bag.

Guiding attention through a virtual room inhabitant(micro-navigation)

Using a geometric model of the room for a distortion free projection

Speech dialogues with objects in the environment(Talking Objects Paradigm)

Embedded speech recognition. Deictic reference triggered by physical action.

Comparison shopping triggered by multimodal interaction

Fusion of intra-, extra-gesture and speech input

Experiment:age-dependent navigation • Investigation of the sensomotoric-cognitive modules related to

navigation• Treadmill-paradigm. Visit in a museum• 32 younger und 32 older Subjects• Varying topologies and navigation aids• Dependent Variables: measured walking distance to target,

landmark knowledge

Lab setting „Old Town“ „City Block“

Results

•Huge gender effect. Possible explanation: Women rely on different representation of space while navigating

•Huge age-effect•Reduction of the senso-motoric load helps older man to improve navigation

Incidental route learning • How does assisted navigation influence learning of our

spatial environment• Which modality should be used to encode navigational

instructions?– Gender differences?– Incremental vs. complete?

• Which technology should be preferred:– PDA-Display– Clip-on

• Dependent Variables:– Route reconstruction– Landmark knowledge

Incidental route learning• Field setting in the Zoo of Saarbrücken• 32 VP (16 m, 16 w)• Wizard of Oz Experiment, Navigational

instructions triggered through a second PDA

Incidental route learningPresentations:

„Hier links“„…“Dependent variables:

Route reconstruction Map reconstruction

Conclusions• Hybrid knowledge representation important

– Embedded into a representation of the userʼs situation• Multi-modal interaction with mobile devices

– Speech and Gesture improve robustness of interaction

• Taking advantage of instrumentation helps to navigate in indoor settings

• Empirical evaluations are necessary to improve spatial appliances (lab and field)

Future work• Improve detection of situations

– Technical constraints– Cognitive constraints (e.g. Bio-sensors)

• Develop framework for instrumented indoor navigation– Mixed usage of private and public devices

• Continue to develop M3I towards a platform to assist spatial cognition (e.g. for the elderly).

Thank you very much

Further information available at:

http://w5.cs.uni-sb.de/real

Ubiquitous Computing - DFKIkrueger/teaching/Website/Ubicomp_2009_2010_files/Ubicomp... · Weiser:...

Documents

Transcript of Ubiquitous Computing - DFKIkrueger/teaching/Website/Ubicomp_2009_2010_files/Ubicomp... · Weiser:...