Towards Virtualization of User Interfaces based on UsiXML J. P. Molina 1,2, J. Vanderdonckt 1, F....

Towards Virtualization Towards Virtualization of User Interfaces of User Interfaces based on UsiXMLbased on UsiXML

J. P. MolinaJ. P. Molina1,21,2, J. Vanderdonckt, J. Vanderdonckt11, F. Montero, F. Montero1,21,2, P. , P. GonzálezGonzález11

11 Université catholique de Louvain Université catholique de Louvain22 University of Castilla-La Mancha University of Castilla-La Mancha

www.usixml.org

Web3D Symposium 2005Web3D Symposium 2005March 29 - April 1 2005, March 29 - April 1 2005, Bangor, UKBangor, UK

March 29 - April 1 2Web3D Symposium 2005, Bangor, UK

ContentsContents OverviewOverview MotivationMotivation Related workRelated work Cameleon reference frameworkCameleon reference framework The Concrete UI level and UsiXMLThe Concrete UI level and UsiXML Mapping from CUI to VRML/X3DMapping from CUI to VRML/X3D VRML97 & X3D toolkitsVRML97 & X3D toolkits A case studyA case study Continuum of User InterfacesContinuum of User Interfaces ConclusionConclusion


OverviewOverview A model-based approach is presented for A model-based approach is presented for

structuring a development process of UIs structuring a development process of UIs based on UsiXML, a XML-compliant based on UsiXML, a XML-compliant language.language.

UsiXML provides a Concrete UI UsiXML provides a Concrete UI description that remains independent description that remains independent from any platform (HTML, Java, VRML).from any platform (HTML, Java, VRML).

To support the rendering of this To support the rendering of this description in a virtual world, two toolkits description in a virtual world, two toolkits have been developed: for VRML97 and have been developed: for VRML97 and X3D worlds.X3D worlds.


MotivationMotivation It is difficult to build a UI that relies solely on It is difficult to build a UI that relies solely on

3D direct manipulation.3D direct manipulation. Some tasks require 1D/2D interaction.Some tasks require 1D/2D interaction. Support for legacy applications is one Support for legacy applications is one

requirement for new 3D desktops.requirement for new 3D desktops. 3D environments must handle 2D very well to 3D environments must handle 2D very well to

break into desktop market.break into desktop market. 3D F/X of UI elements should be rendered in 3D F/X of UI elements should be rendered in

3D, matching much better the mental model of 3D, matching much better the mental model of the user.the user.

There is a need to transform 2D GUIs into 3D.There is a need to transform 2D GUIs into 3D.


Related workRelated work

Automated generation of a 3D UI Automated generation of a 3D UI from its specifications demonstrated from its specifications demonstrated in several domains.in several domains. Maquil’s tool [Maquil, 2004].Maquil’s tool [Maquil, 2004].

Produces 3D dialog boxes describing the UI Produces 3D dialog boxes describing the UI in terms of the StudierStube environment.in terms of the StudierStube environment.

The tool is equipped with an automatic The tool is equipped with an automatic layout facility of the widgets.layout facility of the widgets.



VRML Widgets Working Group.VRML Widgets Working Group. Aimed to define a standard set of widgets.Aimed to define a standard set of widgets. Two different architectures:Two different architectures:

Layered approach [Isaacs & Becker].Layered approach [Isaacs & Becker]. Time control and pointer-surface interaction.Time control and pointer-surface interaction. Interpretation and control of interaction (“feel” of a Interpretation and control of interaction (“feel” of a

widget).widget). Geometry (“look” of a widget).Geometry (“look” of a widget).

Component-based approach [Seidman].Component-based approach [Seidman]. Five components: sensors, internal glue, geometry, Five components: sensors, internal glue, geometry,

external glue and Virtual Reality User Interfaces external glue and Virtual Reality User Interfaces (VRUIs).(VRUIs).



VRML authoring tools.VRML authoring tools. ParallelGraphics Internet Scene ParallelGraphics Internet Scene

Assembler.Assembler. Library of prototypes: button, toggle button, Library of prototypes: button, toggle button,

slider, checkbox, text label, etc.slider, checkbox, text label, etc. Basic set for building HUD (Head-Up Basic set for building HUD (Head-Up

Display) interfaces for PC desktop virtual Display) interfaces for PC desktop virtual worlds.worlds.



X3D-related work.X3D-related work. Project CONTIGRA.Project CONTIGRA.

Defines a XML-based language for the Defines a XML-based language for the specification of widgets, metaphors and specification of widgets, metaphors and interaction techniques.interaction techniques.

X3D language is used to describe the final X3D language is used to describe the final appearance of geometry of each 3D widget.appearance of geometry of each 3D widget.

The set of components has been designed The set of components has been designed taking geometry apart from behaviour.taking geometry apart from behaviour.

Many components have not been developed Many components have not been developed or are not publicly available yet.or are not publicly available yet.



Virtual Reality toolkits.Virtual Reality toolkits. The INQUISITIVE toolkit [Boyd, 1999].The INQUISITIVE toolkit [Boyd, 1999].

Implemented using the MAVERIK toolkit, Implemented using the MAVERIK toolkit, includes window, dial, slider, etc.includes window, dial, slider, etc.

VIEWL [Larimer, 2003].VIEWL [Larimer, 2003]. Based on Qt and DIVERSE, windows are Based on Qt and DIVERSE, windows are

shown as polygons that are tangent to a shown as polygons that are tangent to a sphere that surrounds the user.sphere that surrounds the user.

VRIXML [Cuppens, 2004].VRIXML [Cuppens, 2004]. A XML-based languge for the creation of A XML-based languge for the creation of

2D/3D hybrid interfaces.2D/3D hybrid interfaces.


Cameleon Reference Cameleon Reference FrameworkFramework

Defines UI development steps for multi-Defines UI development steps for multi-context interactive applications [Calvary context interactive applications [Calvary et et al.al., 2003]., 2003]. Final UI (FUI)Final UI (FUI)..

UI running on a particular computing platform (HTML, UI running on a particular computing platform (HTML, Java, VRML).Java, VRML).

Concrete UI (CUI)Concrete UI (CUI).. A mock-up that is independent of any computer platform.A mock-up that is independent of any computer platform.

Abstract UI (AUI)Abstract UI (AUI).. Defines abstract containers and individual components, Defines abstract containers and individual components,

independent of any modality of interaction.independent of any modality of interaction. Task & Concepts (T&C)Task & Concepts (T&C). .

Describe tasks and domain-oriented concepts.Describe tasks and domain-oriented concepts.


Cameleon reference Cameleon reference frameworkframework

Multi-path UI development: Abstraction, Multi-path UI development: Abstraction, Reification and Translation.Reification and Translation.

Task & Concepts

Abstract UI (AUI)

Concrete UI (CUI)

Final UI (FUI)

Task & Concepts

Abstract UI (AUI)

Concrete UI (CUI)

Final UI (FUI)

Context of use A Context of use B

Reification TranslationAbstraction

Task & Concepts

Abstract UI (AUI)

Concrete UI (CUI)

Final UI (FUI)

Task & Concepts

Abstract UI (AUI)

Concrete UI (CUI)

Final UI (FUI)

Context of use A Context of use B

Reification TranslationAbstraction


The CUI level and The CUI level and UsiXMLUsiXML

From a same specification of CUI, a GUI can From a same specification of CUI, a GUI can be obtained in parallel to a virtualization of be obtained in parallel to a virtualization of this GUI.this GUI.

A description of a CUI can be obtained by A description of a CUI can be obtained by successive forward engineering from the successive forward engineering from the T&C level, the AUI level or directly.T&C level, the AUI level or directly.

UsiXML, a XML-compliant User Interface UsiXML, a XML-compliant User Interface Description Language [url: Description Language [url: http://www.usixml.org], provides a CUI http://www.usixml.org], provides a CUI description that remains independent from description that remains independent from any toolkit, whether graphical or virtual.any toolkit, whether graphical or virtual.

This quality is often referred as This quality is often referred as platform platform independenceindependence..



A CUI model consists of a hierarchical A CUI model consists of a hierarchical decomposition of Concrete Interaction decomposition of Concrete Interaction Objects (CIOs).Objects (CIOs).

Each CIO can be sub-typed into sub-CIOs Each CIO can be sub-typed into sub-CIOs depending on the interaction modality depending on the interaction modality chosen: chosen: graphicalCIOgraphicalCIO, , auditoryCIOauditoryCIO, etc., etc.

Every final element of the CUI inherits Every final element of the CUI inherits properties depending on the category they properties depending on the category they belong to.belong to.

The properties chosen for UsiXML belong to The properties chosen for UsiXML belong to the intersection of property sets of major the intersection of property sets of major toolkits and window managers: Windows toolkits and window managers: Windows GDI, Java AWT and Swing, HTML, etc.GDI, Java AWT and Swing, HTML, etc.



Each Each graphicalCIOgraphicalCIO has has specific properties such specific properties such as as isVisibleisVisible, , isEnabledisEnabled, , fgColorfgColor, and , and bgColorbgColor..

Each Each graphicalCIOgraphicalCIO is is sub-typed into one of sub-typed into one of the two possible the two possible categories: categories: graphicalContainergraphicalContainer, and , and graphicalIndividualComgraphicalIndividualComponentponent. .

CUI Model

CIO

graphicalCIO

graphicalContainer graphicalIndividualComponent

CUI ModelCUI Model

CIOCIO

graphicalCIOgraphicalCIO

graphicalContainer graphicalIndividualComponentgraphicalContainergraphicalContainer graphicalIndividualComponentgraphicalIndividualComponent


Mapping from CUI to Mapping from CUI to VRML/X3DVRML/X3D

When mapping a CIO from CUI to a When mapping a CIO from CUI to a VRML or X3D world, different cases VRML or X3D world, different cases are assessed:are assessed: Direct mapping.Direct mapping.

It is not possible to set a one-to-one It is not possible to set a one-to-one mapping, as there is no VRML/X3D mapping, as there is no VRML/X3D primitives that can be matched with a CIO. primitives that can be matched with a CIO. These Web3D languages specify basic These Web3D languages specify basic elements that must be used together to elements that must be used together to create interactive elements.create interactive elements.



When mapping a CIO from CUI to a When mapping a CIO from CUI to a VRML or X3D world, different cases are VRML or X3D world, different cases are assessed:assessed: Direct mapping.Direct mapping. New mapping.New mapping.

There is a need to fill a gap by introducing a new There is a need to fill a gap by introducing a new widget by appropriate implementation.widget by appropriate implementation.

Some attributes defined in the CUI model are not Some attributes defined in the CUI model are not used in the 3D world, while new attributes must be used in the 3D world, while new attributes must be added in order to describe the interface elements added in order to describe the interface elements in the 3D world, such as position and size.in the 3D world, such as position and size.



When mapping a CIO from CUI to a When mapping a CIO from CUI to a VRML or X3D world, different cases VRML or X3D world, different cases are assessed:are assessed: Direct mapping.Direct mapping. New mapping.New mapping. No possible mapping.No possible mapping.

There are CIOs whose VRML/X3D There are CIOs whose VRML/X3D counterpart have not been done yet or are counterpart have not been done yet or are impossible or difficult to implement. An impossible or difficult to implement. An example of this case is the CIO example of this case is the CIO boxbox..



Input/output mapping from desktop-oriented Input/output mapping from desktop-oriented 2D GUI to VRML/X3D virtual world.2D GUI to VRML/X3D virtual world. Display = “Screen” PROTO.Display = “Screen” PROTO.

Rectangular area where screen units (pixels) are Rectangular area where screen units (pixels) are converted into world units (meters).converted into world units (meters).

Serves as a container of UI elements (Serves as a container of UI elements (childrenchildren field). field). Mouse = User’s pointing device.Mouse = User’s pointing device.

VRML/X3D sensors.VRML/X3D sensors. Keyboard?Keyboard?

VRML: requires non-standard extensions (PG’s VRML: requires non-standard extensions (PG’s KbdSensorKbdSensor).).

X3D: X3D: StringSensorStringSensor.. Virtual keyboard.Virtual keyboard.


VRML97 & X3D toolkitsVRML97 & X3D toolkits The architecture for both toolkits is based on:The architecture for both toolkits is based on:

The prototyping mechanism available in both The prototyping mechanism available in both languages.languages.

Some ideas borrowed from the VRML Widget WG.Some ideas borrowed from the VRML Widget WG. For each CIO, there is a final widget implemented For each CIO, there is a final widget implemented

as a PROTO that consists of:as a PROTO that consists of: An interface based on a set of fields.An interface based on a set of fields. Shapes for the geometry and appearance of the widget.Shapes for the geometry and appearance of the widget. Sensors that allow user-widget interaction.Sensors that allow user-widget interaction. Internal logic that manages the state of the widget.Internal logic that manages the state of the widget. Routes that links shapes, sensors and logic.Routes that links shapes, sensors and logic.

Differences between both toolkits are meant to Differences between both toolkits are meant to take advantage of new features of X3D take advantage of new features of X3D specification.specification.


VRML97 & X3D toolkitsVRML97 & X3D toolkits

GUIToggleButton: VRML97 PROTO interface.GUIToggleButton: VRML97 PROTO interface. From attributes specified for From attributes specified for

toggleButtontoggleButton in the CUI in the CUI model.model.

Some UsiXML attributes have Some UsiXML attributes have not a meaning in the toolkit.not a meaning in the toolkit.

Other attributes are added, Other attributes are added, plus plus set_set_ and and _changed_changed fields. fields.

Type of attributes must be Type of attributes must be adapted, e.g.:adapted, e.g.:

booleanboolean = = SFBoolSFBool stringstring? If represents a colour, ? If represents a colour,

SFColorSFColor seems more seems more appropriate.appropriate.

PROTO GUIToggleButton [# USIXML: cio# ...# USIXML: graphicalCio# ...# USIXML: graphicalIndividualComponent# ...# USIXML: toggleButtonfield SFBool defaultState FALSE# VRML97 GUI Toolkit fieldsfield SFInt32 top 0field SFInt32 left 0field SFInt32 width 75field SFInt32 height 25exposedField MFNode label [ ]

eventOut SFTime touchTimeeventIn SFBool set_stateeventOut SFBool state_changed

eventIn SFInt32 set_topeventIn SFInt32 set_lefteventIn SFInt32 set_widtheventIn SFInt32 set_heighteventOut SFInt32 top_changedeventOut SFInt32 left_changedeventOut SFInt32 width_changedeventOut SFInt32 height_changed

GUIToggleButton: X3D PROTO interface.GUIToggleButton: X3D PROTO interface.<ProtoDeclare name="GUIToggleButton"><ProtoInterface> <field accessType="initializeOnly" name="defaultState" type="SFBool" value="false"/><field accessType="inputOutput" name="top" type="SFInt32" value="0"/><field accessType="inputOutput" name="left" type="SFInt32" value="0"/><field accessType="inputOutput" name="width" type="SFInt32" value="75"/><field accessType="inputOutput" name="height" type="SFInt32" value="25"/><field accessType="inputOutput" name="state" type="SFBool"/><field accessType="inputOutput" name="label" type="MFNode"/><field accessType="outputOnly" name="touchTime" type="SFTime"/>

inputOutputinputOutput fields fields were not possible were not possible with VRML97 as with VRML97 as this language does this language does not allow us to not allow us to match them with match them with Script fields.Script fields.

GUIToggleButton: VRML97 vs. X3D GUIToggleButton: VRML97 vs. X3D PROTO.PROTO.PROTO GUIToggleButton [

# USIXML: cio# ...# USIXML: graphicalCio# ...# USIXML: graphicalIndividualComponent# ...# USIXML: toggleButtonfield SFBool defaultState FALSE# VRML97 GUI Toolkit fieldsfield SFInt32 top 0field SFInt32 left 0field SFInt32 width 75field SFInt32 height 25exposedField MFNode label [ ]

eventOut SFTime touchTimeeventIn SFBool set_stateeventOut SFBool state_changed

eventIn SFInt32 set_topeventIn SFInt32 set_lefteventIn SFInt32 set_widtheventIn SFInt32 set_heighteventOut SFInt32 top_changedeventOut SFInt32 left_changedeventOut SFInt32 width_changedeventOut SFInt32 height_changed

<ProtoDeclare name="GUIToggleButton"><ProtoInterface> <field accessType="initializeOnly" name="defaultState" type="SFBool" value="false"/><field accessType="inputOutput" name="top" type="SFInt32" value="0"/><field accessType="inputOutput" name="left" type="SFInt32" value="0"/><field accessType="inputOutput" name="width" type="SFInt32" value="75"/><field accessType="inputOutput" name="height" type="SFInt32" value="25"/><field accessType="inputOutput" name="state" type="SFBool"/><field accessType="inputOutput" name="label" type="MFNode"/><field accessType="outputOnly" name="touchTime" type="SFTime"/>


A case studyA case study

Internet Radio Player.Internet Radio Player. This case study is used as an example This case study is used as an example

running from the topmost level of the running from the topmost level of the reference framework until the bottom level, reference framework until the bottom level, here in VRML97 and X3D.here in VRML97 and X3D.

The following tools are used at different The following tools are used at different levels, all of them based on UsiXML levels, all of them based on UsiXML language:language:

T&C and AUI levelT&C and AUI level: IdealXML, by F. Montero: IdealXML, by F. Montero CUI levelCUI level: GrafiXML, by B. Michotte: GrafiXML, by B. Michotte FUI levelFUI level: VRML97 & X3D toolkits: VRML97 & X3D toolkits www.usixml.org



Task & Concept Model.Task & Concept Model. User’s actions, independently of any User’s actions, independently of any

implementation.implementation.



Abstract User Interface Model (AUI).Abstract User Interface Model (AUI). From the task model, a set of From the task model, a set of

transformations can be applied to transformations can be applied to generate an AUI, later manually generate an AUI, later manually refined.refined.



Concrete Model (CUI).Concrete Model (CUI). New transformations produce a CUI that can New transformations produce a CUI that can

be refined with GrafiXML. Again, everything is be refined with GrafiXML. Again, everything is saved in UsiXML.saved in UsiXML.



Final User Interface (FUI).Final User Interface (FUI). According to given mappings, VRML97 and According to given mappings, VRML97 and

X3D versions of the FUI can be generated X3D versions of the FUI can be generated from UsiXML specifications. The rendered from UsiXML specifications. The rendered FUIs are genuine 3D objects, with real 3D FUIs are genuine 3D objects, with real 3D effects. effects.



Augmenting the GUI with virtual Augmenting the GUI with virtual platform.platform. A virtual A virtual

version of the version of the old old environment environment could be used could be used to run the old to run the old applications applications [Roberts, [Roberts, 2000].2000].


Continuum of User Continuum of User InterfacesInterfaces

Milgram and Kishino [1994] continuum Milgram and Kishino [1994] continuum distributes user interfaces on one distributes user interfaces on one dimension ranging from the dimension ranging from the realreal environment to the environment to the virtualvirtual environment by environment by going through two forms of Mixed Reality going through two forms of Mixed Reality (MR): Augmented Reality (AR) and (MR): Augmented Reality (AR) and Augmented Virtuality (AV).Augmented Virtuality (AV).

We have extended this continuum by adding We have extended this continuum by adding a more continuous range of UIs in the a more continuous range of UIs in the virtual part, and a second dimension to virtual part, and a second dimension to represent the degree of immersion.represent the degree of immersion.


Continuum of User Continuum of User InterfacesInterfaces

Radio Internet PlayerRadio Internet Player

Play

Stop

VolumePS

V

Now playing Volume:74%

Play

Stop

Play

StopP

SV

2D GUI3D renderingof 2D GUI

Digital3D GUI

Virtual3D GUI

Augmentedvirtuality

Augmentedreality

Purereality

Degreeof

immersion

High

Low

Virtuality / digitality


Play

Stop

Volume


Play

Stop

VolumePS

VPS

V

Now playing Volume:74%

Play

Stop

Play

Stop

Play

StopP

SVP

SV

2D GUI3D renderingof 2D GUI

Digital3D GUI

Virtual3D GUI

Augmentedvirtuality

Augmentedreality

Purereality

Degreeof

immersion

High

Low

Virtuality / digitality


Conclusion and further Conclusion and further workwork

UsiXML is a XML-compliant language that UsiXML is a XML-compliant language that allows the designer to describe a UI, whether allows the designer to describe a UI, whether 2D or 3D, graphical or virtual.2D or 3D, graphical or virtual.

It supports a multi-path framework that can be It supports a multi-path framework that can be followed to produce, from a same specification, followed to produce, from a same specification, a GUI and a virtualization of such UI.a GUI and a virtualization of such UI.

The implemented VRML97 & X3D toolkits The implemented VRML97 & X3D toolkits make it possible to render the FUI in a 3D make it possible to render the FUI in a 3D world.world. The “look” is similar to 2D widgets, but the “feel” is The “look” is similar to 2D widgets, but the “feel” is

a genuine 3D behaviour. a genuine 3D behaviour. The continuum of user interfaces has been The continuum of user interfaces has been

extended to accommodate Virtualized User extended to accommodate Virtualized User Interfaces.Interfaces.


Conclusion and further Conclusion and further workwork

With these toolkits, it also possible to With these toolkits, it also possible to simulate in the virtual world the look and simulate in the virtual world the look and feel of the same UI in virtual versions of feel of the same UI in virtual versions of different platforms, such as PDAs, PCs or different platforms, such as PDAs, PCs or even Interactive Worlds.even Interactive Worlds.

Current work deals with supporting the Current work deals with supporting the migration of a UI from one platform to migration of a UI from one platform to another, by visually representing this another, by visually representing this transition.transition.

Thank youThank youMercíMercíGraciasGracias

More info:More info:www.usixml.orgwww.usixml.org

Contact:Contact:[email protected]@info-ab.uclm.es

Towards Virtualization of User Interfaces based on UsiXML J. P. Molina 1,2, J. Vanderdonckt 1, F....

Documents

Transcript of Towards Virtualization of User Interfaces based on UsiXML J. P. Molina 1,2, J. Vanderdonckt 1, F....