Model-driven engineering of multimodal user interfaces
-
Upload
jean-vanderdonckt -
Category
Business
-
view
4.727 -
download
2
description
Transcript of Model-driven engineering of multimodal user interfaces
1 Polytechnic University of Valencia – Valencia, November 30th, 2006
Model-Driven Engineeringof User Interfaces
for Multimodal Web ApplicationsJean Vanderdonckt
Université catholique de Louvain (UCL)Louvain School of Management (LSM)
Information Systems Unit (ISYS)Belgian Laboratory of Computer-Human Interaction (BCHI)
http://www.isys.ucl.ac.be/bchi
2 Polytechnic University of Valencia – Valencia, November 30th, 2006
What is a multimodal user interface?
• Some terminology: notion of mode– We have 5 Modes corresponding to our 5 human senses– The human body has five major senses which operate to gather
information from the world around us, sight, hearing, smell, taste, and touch. Any stimulus to one of the sense areas is detected by sensory nerves and is sent to the brain for interpretation.
3 Polytechnic University of Valencia – Valencia, November 30th, 2006
What is a multimodal user interface?
• Combining several modes simultaneously: some are very hard!
[Tison,2003]
4 Polytechnic University of Valencia – Valencia, November 30th, 2006
What is a multimodal user interface?
• Some terminology: notion of modality– Definition: a particular form of a mode– Example: for auditive mode, we have: speech, music, voice,
chorals, noise– User interface for each mode thus involves one or many
modalities– A system is said to be
• Monomodal iff only one modality is involved• Multimodal iff many modalities are involves
– Some modalities are still in their infancy:• Olfactive user interfaces• Aromatic user interfaces
[Bodnar,2004]
5 Polytechnic University of Valencia – Valencia, November 30th, 2006
What is a multimodal user interface?
• Some terminology: notion of media– Definition: is a hardware/software device allowing interaction
between a user and a system according to a given set of modalities
– Example: for auditive mode, we have several media possible: speech, music, voice, chorals, noise
– User interface for each mode thus involves one or many modalities
[Bellik,2002]
6 Polytechnic University of Valencia – Valencia, November 30th, 2006
Why are multimodal user interfaces important?
• Graphical user interfaces – classical human-computer interface paradigm– Users with limited literacy and typing skills– Handheld devices
• Need for hands and eyes free• Difficult to support mouse and keyboard interaction
– Web access anyplace, anywhere, anytime
• Need for newer, better interface paradigm– Multimodal user interface is a possible answer…– Adaptation to the context of use (user, platform, environment)– More natural and efficient human-computer interaction
[IntuiLab,2006]
7 Polytechnic University of Valencia – Valencia, November 30th, 2006
Why are multimodal user interfaces challenging?
• For Web Engineering– Since presentation and navigation are no longer visual only, how can we
design vocal, tactile and multimodal presentation and navigation within a page or across pages?
• For Model-Driven Approach– So far, little research has been devoted to modelling the various aspects
involved in multimodal web applications. Therefore, there is a need for identifying the appropriate implementation characteristics which will be the turned into high level design options to be supported by model-to-model and model-to-code transformations
• For Human-Computer Interaction:– Developing a user interface for any combination of graphical, vocal, and
tactile interaction modalities remains a challenge because of synchronization issues in time and space
• For Usability Engineering– It is not because we are able to develop a user interface with any
combination of any modalities that the resulting user interface is usable or tractable for a end user. We know little about the usability guidelines to be respected in this area
8 Polytechnic University of Valencia – Valencia, November 30th, 2006
Concepts: The CARE Properties
• From Amodeus European Project (1995!)– Complementarity– Assignment– Redundancy– Equivalence
Put that there
[Bolt,1979]
9 Polytechnic University of Valencia – Valencia, November 30th, 2006
What are the appropriate abstractions?
• Design space that supports effectively and efficiently the progress of the development life cycle towards a final system while assuring some form of quality
• Design space consists of an n-dimensional space where each dimension is denoted by a single design option
• Each design option has a finite set of design values• E.g., label location (left-aligned, centered, right-aligned)
10 Polytechnic University of Valencia – Valencia, November 30th, 2006
Why a design space?
• Advantages:– Clarifies the development process in terms of options– Facilitates the incorporation in the development life cycle of design
options as an abstraction covered by a software– Supports the tractability of more complex design problems
• Shortcomings:– In theory: design options could be very numerous (even infinite)– In practice: impossible to consider a large amount of design options
• Too complex or expensive to implement• Design options do not necessarily address users’ needs and
requirements• Out of the designer’s scope of understanding, imagination or
background• Design decision is not always clear (may violate some usability
principles or guidelines)
11 Polytechnic University of Valencia – Valencia, November 30th, 2006
Multimodal Web Applications
• The languages in which they are implemented (e.g., XHTML, VoiceXML, X+V):– Restrict the amount of possible UIs to obtain– Set directly the CARE properties to Assignment, Equivalence and
Redundancy– Limited to graphic, voice, and tactile modalities
[W3C,2006]
12 Polytechnic University of Valencia – Valencia, November 30th, 2006
Solution space
• Cameleon reference framework• UsiXML language:
– Graphical interaction– Vocal interaction– Multimodal interaction
• Design option-basedtransformational approach:– Step 1: Construct the Task
& Domain Models– Step 2: From Task & Domain to
Abstract User Interface (AUI)– Step 3: From AUI to Concrete User Interface (CUI)– Step 4: From CUI to FUI
[Stanciulescu et al.,2005]
13 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for graphical web UIs
Design options for graphical UIs
Sub-task presentation
Sub-task navigation
Concretization of
navigation & control Navigation
concretizationControl
concretization
14 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for graphical web UIs
• Sub-task presentation: specifies the appearance of each sub-task in the final user interface.
Sub-task presentation
separated
combined
Sub-task 1
Sub-task 2
Sub-task 3
one by one
combined box
tabbed dialog box
floatwindow
Sub-task 1 Sub-task 2 Sub-task 3Sub-task 1
Sub-task 1
Sub-task 2
Sub-task 3
many at once
floatwindow
Sub-task 1
Sub-task 2
Sub-task 3
all in one
areas with separators
group boxes
bulleted list
numberedlist
Sub-task 1
Sub-task 3
Sub-task 2
Sub-task 1
Sub-task 2
Sub-task 3
Sub-task 1
Sub-task 2
Sub-task 3
1. Sub-task 1
2. Sub-task 2
3. Sub-task 3
Abstract level
Concrete level
15 Polytechnic University of Valencia – Valencia, November 30th, 2006
Sub-task presentation – example
• Separated sub-task presentation
T11 T12 T13
T1
AC11
AC12
AC13
AC1
T11 T12 T13
T1
AC11
AC12
AC13
NAC LHS RHS
NAC LHS RHS
• Combined sub-task presentation
16 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for graphical web UIs
• Navigation concretization: identifies the placement and the cardinality of the objects/logically grouped set of objects that ensure the navigation.
Navigation concretization
placement cardinality
local
Sub-task 1
global
Sub-task 1
Sub-task 2
Sub-task 3
simple
Sub-task 2 Sub-task 3Sub-task 1
multiple
Sub-task 2 Sub-task 3Sub-task 1
[Stanciulescu,2006]
17 Polytechnic University of Valencia – Valencia, November 30th, 2006
Sub-task navigation - example
• Local placement for navigation objects
AC1
T11 T12 T13
T1
AC11
AC12
AC13
AIC111 AIC121 AIC131
• Global placement for navigation object
AC1
T11 T12 T13
T1
AC11
AC12
AC13
AIC11
NAC LHS RHS
NAC LHS RHS
18 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for graphical web UIs
• Sub-task navigation: specifies the way in which the dialog control is transferred from one sub-task to another.
Sub-task navigation
asynchronous
Sub-task 1 Sub-task 3Sub-task 2
sequential (synchronous)
Sub-task 1 Sub-task 3Sub-task 2
19 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for graphical web UIs
• Concretization of navigation and control: specifies if the navigation and control are ensured or not by the same object.
Concretization of
navigation & control
separated combined
20 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for graphical web UIs
• Control concretization: specifies the placement and the cardinality of the objects that ensure the control.
Control concretizations
placement cardinality
local
Sub-task 1
global
Sub-task 1
Sub-task 2
Sub-task 3
multiple
Sub-task 2 Sub-task 3Sub-task 1
simple
Sub-task 2 Sub-task 3Sub-task 1
21 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for multimodal web UIs
• Consider the ergonomic criteria for the evaluation of human-computer interfaces– Compatibility, consistency, work load, adaptation, dialog control,
guidance, error management
• Associate a corresponding CARE property to each design value.
Design options for multimodal UIs
Prompt Grouping /Distinction of items
Immediatefeedback Guidance
[Scapin & Bastien,1997][Vanderdonckt,1995]
22 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for multimodal web UIs
• Prompt: refers to the interaction channels available in order to lead the users to take specific actions whether it be data entry or other tasks.– vocal (A)– graphical (A)– multimodal (E, C, R)
23 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for multimodal web UIs
• Grouping/Distinction of items: concerns the organization of information in relation to one another.– Input: any information input from the user to the system
• vocal (A)• graphical (A)• multimodal (E, C, R)
– Output: any information output from the system to the user• vocal (A)• graphical (A)• multimodal (E, C, R)
24 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for multimodal web UIs
• Immediate feedback: concerns the system responses to users’ action– vocal (A) – graphical (A)– multimodal (E, C, R)
25 Polytechnic University of Valencia – Valencia, November 30th, 2006
Design options for multimodal web UIs
• Guidance: refers to the means available to advise, orient, inform, instruct and guide the users throughout their interactions with the system– Guidance for input: any guidance offered to the user in order to
guide him with the input• Textual (A), Iconic (A), Acoustic (A), Speech (A)• Multimodal (E, C, R)
– Guidance for output: any guidance offered to the user in order to guide him with the feedback
• Textual (A), Iconic (A), Acoustic (A), Speech (A)• Multimodal (E, C, R)
26 Polytechnic University of Valencia – Valencia, November 30th, 2006
Multimodal example
• Multimodal text field– Prompt: multimodal (R)– Grouping for input: multimodal (E)– Immediate feedback: multimodal (R)– Guidance:
• Input: iconic (A)• Output: iconic (A)
Prompt: multimodal
Please say your name
(vocal +graphical)1
Input: multimodal
Juan Gonzalez
2
(vocal +graphical)
Juan Gonzalez
Guidance for input: iconic
Guidance for feedback: iconic
Feedback:multimodal
Your answer was: Juan Gonzalez
3
(graphical + vocal)
[Stanciulescu,2006]
27 Polytechnic University of Valencia – Valencia, November 30th, 2006
Vocal Concrete Interaction Objects
• VocalContainers:– VocalGroup– VocalForm– VocalMenu– VocalConfirmation
• VocalIndividualComponents:– VocalOutput:
• VocalFeedback• VocalPrompt• VocalMenuItem
– VocalInput– VocalNavigation– Break– Exit
• EventTypes: error, help, noInput, noMatch
28 Polytechnic University of Valencia – Valencia, November 30th, 2006
Vocal Concrete Interaction Objects
• Example: virtual ice cream shop
29 Polytechnic University of Valencia – Valencia, November 30th, 2006
Multimodal Concrete Interaction Objects
• MultimodalCIO: combination of graphical CIO and vocalCIO•
outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance
groupBox + radioButtons
vocalMenu + vocalItems + vocalInput
groupBox + radioButtons + vocalMenu + vocalItems +
vocalInput + guidance
Graphical Vocal Multimodal
30 Polytechnic University of Valencia – Valencia, November 30th, 2006
Concrete UI Relationships
• GraphicalRelationships– GraphicalTransition– GraphicalAdjacency– GraphicalContainment
• VocalRelationships– VocalTransition– VocalAdjacency– VocalContainment
• Synchronization: between vocalCIOs and graphicalCIOs
31 Polytechnic University of Valencia – Valencia, November 30th, 2006
Tool support
• MultimodaliXML – five softwaremodules:– IdealXML– TransformiXML– GrafiXML– CFB Generator– XHTML+Voice Generator tool
32 Polytechnic University of Valencia – Valencia, November 30th, 2006
Multimodal Concrete Interaction Objects
• Syntax example: multimodal text field • with equivalent input and redundant output
33 Polytechnic University of Valencia – Valencia, November 30th, 2006
Specification of transformation
• Graph-based patterns• Syntactically typed patterns• Textual concrete syntax of patterns: UsiXML syntax
34 Polytechnic University of Valencia – Valencia, November 30th, 2006
How transformation rules work…
• LHS: Find an occurrence of LHS into G• NAC: Check that NAC does not match into G. If there is a
match then skip to another occurrence of LHS • RHS: Replace LHS by RHS
Transformation rule
Initial Model
Resulting Model
[Limbourg,2004]
35 Polytechnic University of Valencia – Valencia, November 30th, 2006
UsiXML syntax of transformation rules
NAC
NAC
RHS
mappings
36 Polytechnic University of Valencia – Valencia, November 30th, 2006
Transformation rule
NAC:LHS:
RHS:
Multimodal prompt
Guidance for input
Multimodal input
Guidance for output
Vocal feedback
37 Polytechnic University of Valencia – Valencia, November 30th, 2006
Case studies
• 2 case studies: Virtual Polling system and Car rental system
38 Polytechnic University of Valencia – Valencia, November 30th, 2006
Conclusion
• Model-driven Engineering of Multimodal Web applications is possible
• Requirements fulfillment:– Support for multimodal input/output– CARE properties support for input/output modalities– Approach based on design space composed of 13 design options– Design space covers a subset of the total number of UIs which
can be developed with X+V
• Methodology covers a subset of the UIs which can be developed with X+V
39 Polytechnic University of Valencia – Valencia, November 30th, 2006
Conclusion
• Advantages of combining design options into a design space:
Vertues of a design space
– Descriptive– Generative– Comparative
[Stanciulescu & Vanderdonckt,2006]
[Beaudouin-Lafon,2004]
40 Polytechnic University of Valencia – Valencia, November 30th, 2006
Future work
• Investigate the implementation of a new transformation engine: any suggestion?
• Reinforce existing vocal components• Extend the ontology with new vocal components• Design space improvement• Extend transformation catalogs• Adaptation of multimodal web user interface to the context of
use
41 Polytechnic University of Valencia – Valencia, November 30th, 2006
References
• See videos and papers on MultimodaliXML at:http://www.usixml.org/index.php?view=page&idpage=51
• Stanciulescu, A., Vanderdonckt, J., Design Options for Multimodal Web Applications, Proc. of 6th Int. Conf. on Computer-Aided Design of User Interfaces CADUI’2006 (Bucharest, 6-8 June 2006), Chapter 4, Springer-Verlag, Berlin, 2006, pp. 41-56
• Stanciulescu, A., Limbourg, Q., Vanderdonckt, J., Michotte, B., Montero, F., A Transformational Approach for Multimodal Web User Interfaces based on UsiXML, Proc. of 7th ACM Conf. on Multimodal Interfaces ICMI’2005 (Trento, 4-6 October, 2005), ACM Press, New York, 2005, pp. 259-266
42 Polytechnic University of Valencia – Valencia, November 30th, 2006
Thank you very much for your attention
For more information and downloading,http://www.isys.ucl.ac.be/bchi
http://www.usixml.orgUser Interface eXtensible Markup Language
http://www.similar.ccEuropean network on Multimodal UIs
Special thanks to all members of the team!