Adaptation and Continuity in Multi-Device Environments

1

Adaptation and ContinuityAdaptation and Continuityi M ltii M lti D i E i tD i E i tin Multiin Multi--Device EnvironmentsDevice Environments

Fabio Paternò

HIIS Laboratory ISTI-C.N.R.Pisa, Italy

http://giove.isti.cnr.it/

Tutorial Tutorial GoalsGoals How to obtain interfaces able to adapt to multiple

device types while preserving usability? State of art in terms of approaches design criteria State of art in terms of approaches, design criteria,

tools Particular attentiont to adaptation in Web applications

and use of model-based techniques Understanding the space of the possible solutions in

order to better apply them and think about new

2

pp ysolutions

Consider how to address the device adaptation issueboth at design time and at run-time

Discussion on how adaptation and continuity can besupported in migratory user interfaces

2

StructureStructure

Introduction, Basic Concepts, Issues Usability and Task in Multi-device environments Usability and Task in Multi device environments Authoring Multi-Device Interfaces Model-based Support for Multi-device Interfaces Tools for Desktop-to-Mobile Adaptation Adaptation Customization Migratory Interfaces

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –http://hiis.isti.cnr.it 3

Migratory Interfaces Partial / Trans-modal Migration Conclusions and Discussion

WhyWhy AdaptationAdaptation toto the the ContextContext ofof UseUse

User:

• Preferences

Environment:

•Position

• Light

• Noise, …

• Social aspects

• Knowledge

• Goals

• Background

•…

Device:

4

•Privacy

•Collaboration

•…….

• Screen resolution

• Connectivity

• Browser

• …

3

Display Display SizeSize and and ResolutionResolution

Personal Computer (PC) usually varies between800x600 and 1920x1200 pixels,Mobile devices usually between 240x240 and Mobile devices usually between 240x240 and960x640 pixels (Iphone 4)

Simple phones have even lower resolutions It varies more with mobile devices than desktop

ones The Moore Law continuously changes these

b !

5

numbers!

Mobile Mobile InteractionInteraction TechniquesTechniques VariationVariationhttp://www.w3.org/TR/http://www.w3.org/TR/mwabpmwabp/#bp/#bp--presentationpresentation--interactioninteraction

Focus Based: The browser focus moves through elements The current focus of the page is easily determined because the focus element will be

highlighted;Focus area can move from one selectable element to another (e g from link to link) even when Focus area can move from one selectable element to another (e.g. from link to link) even when widely spaced

Pointer Based: Key-based navigation controls a pointer that can cover any part of the screen Selectable elements that are associated with each other need to be close as moving the pointer

can be slow; Selectable elements need to be large enough to be easily selected -- since the pointer often

moves in steps of between 5 - 10 pixels; Selectable elements should have rollovers to make it clear when the pointer has entered their

active area.

Touch Based: Events are related directly to a finger or stylus


Touch Based: Events are related directly to a finger or stylus touch position on the screen. Selectable elements may be widely spaced since the user can select them directly; Selectable elements must be large enough to be easily selected (e.g. list items should have a

height of at least 30px); No elements are in focus until they are selected so extra information cannot be passed to the

user (e.g. rollovers will not work).

4

Usability in Mobile InteractionUsability in Mobile Interaction Minimize text input Exploit the elements of the mobile device, such

as accesskeysy Consistency between platforms Prevent user error The purpose of the interface elements should be

clear Avoid overloading the user interface with many

elements


elements Limit the need for scrolling Short time access Access to small pieces of information

SupportingSupporting TasksTasks in in MultiMulti--platformplatform EnvironmentsEnvironments

Platform definition Same task on multiple platforms in the samep p

manner Same task on multiple platforms but with

different user interface elements Same main task, with different levels of

subtasks


Dependencies among tasks performed on different platforms

Tasks meaningful only on some platform types

5

SameSame task on multiple task on multiple platformsplatformswithwith differentdifferent useruser interfacesinterfaces


Task Task MeaningfulMeaningful onlyonly on on Some Some PlatformsPlatforms

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –http://hiis.isti.cnr.it

10

Search for flightsPromotionsReservations

6

Authoring MultiAuthoring Multi--device device InterfacesInterfaces

Platform specific authoring (e g Platform-specific authoring (e.g. Amazon)

Multiple-device authoring (e.g. with CSS)

Single authoring


Single authoring Automatic re-authoring

Multiple Multiple DeviceDevice AuthoringAuthoringDamaskDamask

(James (James LinLin PhDPhD –– CHI’08)CHI’08)•Sketches

•Layers•Layers

•Patterns


7

Automatic ReAutomatic Re--AuthoringAuthoring Scaling, such as Safari on IPhone

T d i t l t l t i t Transducing, translates elements into other formats, and compresses and converts images to match device characteristics, such as Mowser or Skweezer, T f i f th t dif b th


13

Transforming goes further to modify both contents and structures originally designed for desktop systems to make them suitable to display on small screens

Transforming (examples)Transforming (examples) Single column, (for example Opera SSR)

eliminates scrolling in one dimension, itgreatly increases the amount of scrolling in g y gthe other dimension.

Fisheye (for example Fishnet) is a fisheyeWeb browser that shows a focus region at a readable scale while spatially compressingpage content outside the focus region

Overview + detail splits a Web page intol i l i d id i


multiple sections and provides an overviewpage with links to these sections.

8

NarrowNarrow SolutionSolution (e.g. Opera SSR)(e.g. Opera SSR)

15………. A lot of vertical scrolling !!!!

NarrowNarrow SolutionSolution The order for the content follows that of the

markup file starting with the top The images are scaled to the size of the

screen The text is always visible and the content is

compacted without blank spaces Content that requires space such as maps

16

Content that requires space such as mapsand tables can become unreadable

Sometimes it is difficult to understand thatthe page has changed because the initialpart is the same

9

RequirementsRequirements forfor the Nokia the Nokia SolutionSolution (CHI 2006, Roto (CHI 2006, Roto etet al.)al.)

Remove the need of horizontal scrolling to read Remove the need of horizontal scrolling to readtext

Provide enough contextual information to give anidea of the page structure and the currentlocation in it

Don’t destroy the original page layout


Don’t introduce modal interaction

MiniMap MethodMiniMap Method(CHI 2006, Roto et al.)(CHI 2006, Roto et al.)

Nokia S60 phones Overview + Detail


Original Compacted Mini-Map

10

ApplicationApplication ofof Information Information VisualizationVisualization TechniquesTechniques

Fisheye calendarallows complextasks to be completed more quickly

19

Focus + ContextIn DateLens[Bederson et al 02, 04]

ModelModel--based approachesbased approaches Allow designers and developers to concentrate

on main semantic aspectsL th t t h t Languages that represent such aspects

Avoid need to learn and manage manyimplementation languages

Linking semantic information and implemementation elements


p Interoperability through many possible

implementation languages Facilitate support of assistive technology

11

Abstraction Levels Abstraction Levels in Interactive Systemsin Interactive Systems

Task and object – Activity oriented Task and object Activity oriented – I want to select a work of art

Abstract Interface – Platform Independent – Single selection object with high cardinality

Concrete Interface –Platform Dependent – List Interaction object with X elements


List Interaction object with X elements Implementation

– List object in Java or XHTML or ....

The The ConcurTaskTreesConcurTaskTreesNotationNotation forfor Task Task ModelsModels

Hierarchical structure

Temporal relations


22

Task Allocation

12

Support Support forfor ApplicationsApplicationsbasedbased on Web Serviceson Web Services(http://giove.isti.cnr.it/tools/Mariae)(http://giove.isti.cnr.it/tools/Mariae)

Task ModelTask Model

Web Services

Annotations

Desktop AUI

Specification

PDA AUI

Specification

Multitouch

Phone AUI

Specification

Multitouch

Phone CUI PDA CUI

Specification

Desktop CUI

Specification

Desktop AUI

Specification

PDA AUI

Specification

Multitouch

Phone AUI

Specification

Multitouch

Phone CUI PDA CUI

Specification

Desktop CUI

Specification

TMLanguage

TMLanguage

AUI Language

PDA CUI Desktop

M‐touch Phone CUI

Language

AUI Language

PDA CUI Desktop

M‐touch Phone CUI

Language

MARIA

SpecificationSpecificationSpecification

SpecificationSpecificationSpecification

Specifications Languages

Languagep

CUI Language

LanguageLanguage

pCUI

Language

Language

Reverse Reverse EngineeringEngineeringAutomatic Automatic ModelModel CreationCreation !!

24


13

AdaptationAdaptation in in AuthoringAuthoring EnvironmentsEnvironments(SUPPLE, (SUPPLE, GajosGajos and and WeldWeld))

Takes a functional specification of the interface, the device-specific constraints, a typical usage trace, and a cost function.

The cost function is based on user preferences and expected The cost function is based on user preferences and expected speed of operation

SUPPLE’s optimization algorithm, finds the user interface which minimizes the cost function while also satisfying all deviceconstraints.

Adaptation and Continuity in Multi-Device Environments 25

Application server-side adaptation

ArchitecturalArchitectural SolutionsSolutions forforAutomatic Automatic Web Web AdaptationAdaptation

Server Proxy Client

Server Client

Request

Content

Request/Device capability

Request/ Device capability

Adapted content

Proxy-side adaptation

Adapted Content


Server ClientRequest

Content Adapted content

Client-side adaptation

14

DynamicDynamic UserUser Interface Interface AdaptationAdaptation

Use of Logical Description Languages Existing Web desktop Applications Automatic user interface generation


ParametricParametric BidimensionalBidimensionalCostCost--basedbased SemanticSemantic RedesignRedesign

Font adjustments accordinguser preferences

ImagesInteractor replacementLong texts

GroupingRelationData tablesLayout tables

Paternò, Zichitella, HCSE 2010

15

Adaptation Solutionshttp://mowser.com http://www.skweezer.com

•Uses predefined style sheets•No support for tables or long texts

•Limited reduction of image dimensions•Aims to reduce horizontal scrolling

Customizing Adaptation

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 30

16

Customizing Adaptation

31

BrowsingBrowsing LargeLarge TableTable in in SmallSmall ScreensScreens

Ohnishi and Tajima, UIST’08

32Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile

HCI 2010 – http://hiis.isti.cnr.it

Ease comparison of cells far from each other

17

PagePage SummarizationSummarization

The Abstraction-based approach uses sentencemanipulation techniques like reductionmanipulation techniques like reduction, compression and reformulation.

The Extraction-based approach assigns scores to sentences in order to select those which better represents the whole text Feature based (e.g. term frequency, sentence

position, attributes…); machine learning, graph


p , ); g, g pbased techniques

MicroMicro--levellevel SummarizationSummarization((PowerBrowserPowerBrowser, , BuyukkoktenBuyukkokten etet al.)al.)

The importance of a keyword depends on the frequency it occurs in a text and in a largercollection

A word within a given text is considered most important if it occurs frequently within the text, but infrequently in the larger collection

The significance factor of a sentence is derivedfrom an analysis of its constituent words

The sentences in which the greatest number ofe se te ces c t e g eatest u be ofrequently occurring distinct words are found in closest proximity to each other are probablyimportant

MEAD is a public multi-document summarizationsystem, which provides more flexible support in thisarea (see http://www.summarization.com/mead/)

18

Adaptation in MultiAdaptation in Multi--User and User and MultiMulti--Device environmentsDevice environments

WebSplitter (Han et al., CSCW’00)- collaborative Web browsing by creating personalized partial views of the same Web page depending on the user and the device. Developers have to specify the Web content in XML and define a policy file indicating what content tagscontent in XML and define a policy file indicating what content tags should be shown for each device and user.

Concepts and Tools for Multi-Device User Interfaces 35

HowHow People People UseUseMultiple Multiple DevicesDevices ((DearmanDearman & Pierce ‘08) & Pierce ‘08)

A recent study in US of 27 people from academic and industrial research revealed that on average they employ more than five computing devicesemploy more than five computing devices

Associating a user’s activities with a particular device is problematic for multiple device users because many activities span multiple devices

Device use varies by user and circumstance; users assign different roles to devices both by choice and by constraint. Users employ a variety of techniques for accessing Users employ a variety of techniques for accessing information across devices but participants reported managing information across their devices as the most challenging aspect of using multiple devices.

36

19

Flexible Access in MultiFlexible Access in Multi--Device environmentsDevice environments

Moving objects: across interactived i th h i k d ddevices through pick-and-drop

Distributed user interfaces: applicationlogic receiving input from multiple devices

Migratory user interfaces: change


g y gdevice, interface migration with state preservation

PickPick--andand--dropdropJun Rekimoto, UIST’97 / CHI’98Jun Rekimoto, UIST’97 / CHI’98

38

A Direct-manipulation Technique for Multiple-Computer Environments

20

PUC PUC (Personal Universal Controller) [CMU, Nichols, Myers](Personal Universal Controller) [CMU, Nichols, Myers]

Personal MobileDeviceAutomatically

G t d

AbstractControl

GeneratedInterface

39

AbstractSpecification State Feedback

Why Migratory InterfacesWhy Migratory Interfaces Our life is becoming a multi-device experience

One of the main source of frustration is that we One of the main source of frustration is that weneed to restart for each device change

Need for continuous access to interactiveservices across various devices

Migratory user interfaces can transfer among different devices (from ‘source’ devices to


40

(‘target’ devices), so as to allow the users to continue their tasks

Application domains such as shopping, bids for auction on line, games, making reservations

21

MigrationMigration PhasesPhases Device Discovery: Devices notify their presence When to Migrate: Migration Trigger (User or

System or Mixed initiative)System or Mixed initiative) Where to Migrate: Identification of Target and

its resources What to Migrate: User interface and/or

Application logic How to Migrate: Adaptation (depending on the

type of target)

41

type of target) State Persistence: Source state extraction and

association to target version Activation in the target device: Upload the

adapted version at the point in which the user leftoff on the source device

UsabilityUsability in in MigrationMigration Relevant aspects to continuity

Timed b h Time required by the migration process

Time from the last interaction in the source device

Adaptation process When adaptation makes unclear how to continue the task

Predictability: The target device

42

The target device Which part of the user interface migrate Where the result of an interaction will be presented

Learning: to get familiar with the migration process

22

What is the state to preserveWhat is the state to preserve User Input Focus Cookies Session History Bookmarks

J S i t

43

JavaScripts

Task: SelectMenuvalue: Vegetable

Your SelectedMenu is Vegetable

Web Web SessionSession MigrationMigration UsingUsingDynamicDynamic 2D 2D BarcodesBarcodes

A. Alapetite, PUC 14(1): 45-52 (2010)

Append the ID of the current session (token) to the current URL query string

When the Web server receives a request from a new device (the mobile phone)


with a known existing session ID, the server will provide the same content as the one served to the previous device (the desktop).

23

ApproachesApproaches toto MigrationMigration Bharat and Cardelli (UIST’95) - the migration of entire

applications

Ch d D (UIST’96) h i ti i Chung and Dewan (UIST’96) - when migration is triggered the environment starts a fresh copy of the application process in the target system, and replays the saved sequence of input events to the copy. This solution does not support interface adaptation.

Kozuch and Satyanarayanan (2002) - solution for migration based on the encapsulation of all volatile

45

migration based on the encapsulation of all volatile execution state of a virtual machine (migration of applications among desktop or laptop systems)

Melchior, Grolaux, Vanderdonckt (EICS’09) - solution for distributed user interfaces with extension of Tcl/Tk


Application

• User Interface

Device 1

Mi ti F k

Architectural Framework Architectural Framework

Migration Platform

User Interface

• Application Logic

• Connection to OPEN Client

OPEN Client

R i b k d

Migration Framework

•Server

Migration Services Orchestration

Migration Services

46

• Runs in background

• Connects to OPEN platform framework

gUser Interface migration

Application logic migrationState persistence

Context managementTrigger management

24

Device discovery and selectionDevice discovery and selectionTarget Device Selection

•same environment

•device availability

•same user (or shared)

•interaction features


Example representation of the migration environment

DeviceDevice SelectionSelection

Ghiani, PhD


25

Migration Trigger (6)Migration

Client

Device Discovery (1)

ExampleExample ArchitectureArchitecture MigrationMigration

Request Page (2)

Upload (8)

Provide Annotated Page (5) Proxy Server

ReverseSemantic Redesign

State MapperGenerator

Client

Cookies

DesktopApplication

49

Trasmission of DOM + current state through callback AJAX (7)

Request Page (3)Provide Page (4)

Application Server

PDA

Migrating between mobile Migrating between mobile and Digital Tvand Digital Tv

Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 – http://hiis.isti.cnr.it 50

26

ExampleExample MigrationMigration (Video)(Video)(shopping scenario)(shopping scenario)


TransTrans--ModalModal MigrationMigration

Vocabulary 1kMay 25 Last minute offerLuois

SizeSizeDate Date &Time&Time

SubjectSubjectSenderSender

1kMay 25 Last minute offerLuois

SizeSizeDate Date &Time&Time

SubjectSubjectSenderSender

y Information organization Different concrete objects

Not a simple one to one mapping

1kJune 7 10:00

Re: Cameleonmeeting

Mary Rex

2kJune 5 12:20

Learn about JavaLuoisLestat

3kJune 3 11:02

Re: Cameleonmeeting

GeorgeClow

1kMay 25 9:23

Last minute offerLuoislestat

1kJune 7 10:00

Re: Cameleonmeeting

Mary Rex

2kJune 5 12:20

Learn about JavaLuoisLestat

3kJune 3 11:02

Re: Cameleonmeeting

GeorgeClow

1kMay 25 9:23

Last minute offerLuoislestat

52

Not a simple one to one mapping Task not Supported Task implemented with different concrete

objects

Adaptation of runtime data

27

(grouping sound) Please say your name.

Please say your surname.

Differences in task support Differences in task support implementationimplementation

y y

Please say the date of reservation.

Your name is …. Your surname is …. You have booked a table on … and You prefer…

What type of menu do you prefer: fish; meat; vegetable (grouping sound)

53

(grouping sound) If you want to cancel the reservation , say delete; If you want to confirm the reservation, say ok (grouping sound).

Grouping OperatorGrouping Operator Object that Support the taskObject that Support the task“Provide date”“Provide date”

Adaptation of runtime dataAdaptation of runtime data

(grouping sound) Please say your name.

Your name is LouisLouis. You have booked a table on 3 July3 July.

Name: LouisDate: 3 July

Please say your surname.

Please say the date of reservation.

Your name is …. Your surname is …. You have booked a table on and You

What type of menu do you prefer: fish; meat; vegetable (grouping sound)

54

You have booked a table on … and You prefer…

(grouping sound) If you want to cancel the reservation , say delete; If you want to confirm the reservation, say ok (grouping sound).

28

ApplicationApplication LogicLogic ReconfigurationReconfiguration

ClausthalUniversity in OPEN Project

Partial Migration(Mobile HCI 2010 paper)

56

29

Partial Migration

PartialPartial MigrationMigration withwith Interface Interface StructureStructure RepresentationRepresentation

30

PartialPartial MigrationMigration withwithDirectDirect SelectionSelection

59

PartialPartial MigrationMigration withwithDirectDirect SelectionSelection (video)(video)

60Fabio Paternò - Adaptation and Continuity in Multi-Device Environments – Mobile HCI 2010 –http://hiis.isti.cnr.it

31

Research AgendaResearch Agenda Automatic Desktop-to-Vocal Adaptation Migration involving multiple usersg g p Integration of wider set of interaction modalities

in migratory environments Migration from multiple devices to multiple

devices Authoring Environments for Distributed User

I t f


61

Interfaces Privacy and Security of Migratory User Interfaces Applying Machine Learning Techniques to User

Interface Adaptation

More Info atMore Info at

EU OPEN Project http://www.ict-open.eu

EU SERENOA Project htt // f 7 / EU SERENOA Project http://www.serenoa-fp7.eu/

W3C group on model-based interfaceshttp://www.w3.org/2005/Incubator/model-based-ui/charter/

EU Artemis SMARCOS Project http://www.smarcos-project.eu/

F th i S i B k Mi t Forthcoming Spinger Book on MigratoryInteractive Applications in UbiquitousEnvironments


62

32

Don’t Don’t ForgetForget EICS 2011 !EICS 2011 !


63

Adaptation and Continuity in Multi-Device Environments

Documents

Transcript of Adaptation and Continuity in Multi-Device Environments