A Transformational Approach for Developing Multimodal Web User

Interfaces

Adrian Stanciulescu

Plan

Context Motivations Thesis statement Requirements for multimodal web UIs Proposal

Reference framework for multimodal web UIs Design space for multimodal web UIs Transformational method for producing multimodal UIs Tool support

Case studies Conclusion & Future work

Context

Graphical user interfaces – classical human-computer interface paradigm Users with limited literacy and typing skills Handheld devices

Need for hands and eyes free Difficult to support mouse and keyboard interaction

Web access anyplace, anywhere, anytime

Need for newer, better interface paradigm Multimodal user interface is the answer… Adaptation to the context of use (user, platform, environment) More natural and efficient human-computer interaction

Motivations

Modality interaction flexibility Faster interaction Lower incidence of errors and esier error recovery True device mobility Usability improvement Robust systems Enhancement of device effectiveness Improved experience Lack of multimodal applications

Thesis statement

Development of multimodal web user interfaces Three types of user interfaces:

Graphical Vocal Multimodal

Conceptual and methodological aspects for a model-to-model transformational approach

Development life cycle based on a design space composed of design options

Requirements

Support for multimodal input and output CARE properties support for input and output

modalities Approach based on design space Openness to new modalities Separation of modalities

Requirements (cont’d)

Ontological independence of modalities Transformation-based development Machine processable Human readable Ontological homogeneity Reuse of specification Methodological explicitness Methodological extendibility Support for tool interoperability

Reference framework

Camelon reference framework UsiXML – selected language:

Structured according to four levels of abstraction

Progressive transformation from Task & Concept level to Final UI Support for the extension of the transformational methodology (graph transformation) Underlying unique formalism (graph-based syntax) Support for reusability of specification and transformations thanks to

the XML structure Support for modality independence Allows introduction of new interaction modality Cross-toolkit development

Vocal Concrete Interaction Objects

Shortcomings of UsiXML previous version:

No specialized containers for user-system dialog No specialized container for multiple selection No differentiations between the system’s output types No object dedicated to system interruptions

Vocal Concrete Interaction Objects (cont’d)

VocalContainers: VocalGroup VocalForm VocalMenu VocalConfirmation

VocalIndividualComponents: VocalOutput:

VocalFeedback VocalPrompt VocalMenuItem

VocalInput VocalNavigation Break Exit

EventTypes: error, help, noInput, noMatch

Vocal Concrete Interaction Objects (cont’d)

Example: virtual ice cream shop

Multimodal Concrete Interaction Objects

MultimodalCIO: combination of graphical CIO and vocalCIO

outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance

groupBox + radioButtons

vocalMenu + vocalItems + vocalInput

groupBox + radioButtons + vocalMenu + vocalItems +

vocalInput + guidance

Graphical Vocal Multimodal

Concrete UI Relationship

GraphicalRelationships GraphicalTransition GraphicalAdjacency GraphicalContainment

VocalRelationships VocalTransition VocalAdjacency VocalContainment

Synchronization: between vocalCIOs and graphicalCIOs

Multimodal Concrete Interaction Objects

Syntax example: multimodal text field

with equivalent input and redundant output

Specification of transformation

Graph-based patterns Syntactically typed patterns Textual concrete syntax of

patterns: UsiXML syntax

How transformation rules work…

LHS: Find an occurrence of LHS into G NAC: Check that NAC does not match

into G. If there is a match then skip to another occurrence of LHS

RHS: Replace LHS by RHS

Transformation rule

Initial Model

Resultant Model

UsiXML syntax of transformation rules

NAC

NAC

RHS

mappings

The four steps of the transformational approach

Step 1: Construct the Task&Domain Models

Step 2: From Task & Domain to AUI

Step 3: From AUI to CUI Step 4: From CUI to FUI

Sub-steps of the transformational approach

Design space for web UIs

Support for the progressive development life cycle towards a final system

Composed of a set of design options Each design option has a finite set of design values Advantages:

Clarifies the development process in terms of options Facilitates the incorporation in the development life cycle of

design options as an abstraction covered by a software Supports the tractability of more complex design problems

Design options for graphical web UIs

Design options for graphical web UIs

Design options for multimodal web UIs

Prompt: vocal (A), graphical (A), multimodal (E, C, R) Grouping/distinction of items:

Input: vocal (A), graphical (A), multimodal (E) Output: vocal (A), graphical (A), multimodal (E, C, R)

Immediate feedback: vocal (A), graphical (A), multimodal (E, C, R)

Guidance: Input: textual (A), Iconic (A), Acoustic (A), Speech (A),

Multimodal (E, C, R) Immediate feedback: textual (A), Iconic (A), Acoustic (A),

Speech (A), Multimodal (E, C, R)

Step 2: From Task & Domain to AUI

Separated sub-task presentation

Combined sub-task presentation

T11 T12 T13

T1

AC11

AC12

AC13

AC1

T11 T12 T13

T1

AC11

AC12

AC13

NAC LHS RHS

NAC LHS RHS

Step 3: From AUI to CUI

Multimodal text field Prompt: multimodal (R) Input: multimodal (E) Immediate feedback: multimodal (R) Guidance:

Input: iconic (A) Output: iconic (A)

Prompt: multimodal

Please say your name

(vocal +graphical)1

Input: multimodal

Juan Gonzalez

2

(vocal +graphical)

Juan Gonzalez

Guidance for input: iconic

Guidance for feedback: iconic

Feedback:multimodal

Your answer was: Juan Gonzalez

3

(graphical + vocal)

Transformation rule

NAC: LHS:

RHS:

Multimodal prompt

Guidance for input

Multimodal input

Guidance for output

Vocal feedback

Case studies 2 case studies: Virtual Polling system and Car rental system

Tool support

IdealXML TransformiXML GrafiXML CFB Generator Communication Flow

Builder XHTML+Voice

Generator tool

Conclusions

Model-drive approach for the development of multimodal web user interfaces

Requirements fulfillment: Support for multimodal input/output CARE properties support for input/output modalities Approach based on design options

Methodology covers a subset of the UIs which can be developed with X+V

Conclusions Advantages of combining design options into a design space:

Descriptive Generative Comparative

Conclusions

Work positioned at the intersection of SE and HCI Focus on the feasibility of code generation (2 case

studies) Usability point of view:

Inexistent published studies on multimodal UI design in SE and HCI

Few ongoing works caused by the reduced number of multimodal applications

Internal validation of the Ph.D. (study based on qualitative principles derived from the cognitive psychology)

Future work

Investigate the implementation of a new transformation engine: MOF QVT

Reinforce existing vocal components Extend the ontology with new vocal components Design space improvement Extend transformation catalogs Adaptation of multimodal web user interface to

the context of use