Interactive Design of Multimodal User Interfaces -...

Journal on Multimodal User Interfaces / SpringerDOI 10.1007/s12193-010-0044-2

Interactive Design of Multimodal User InterfacesReducing technical and visual complexity

Werner A. Konig · Roman Radle · Harald Reiterer

Received: 24 July 2009 / Accepted: 22 January 2010

Abstract In contrast to the pioneers of multimodal in-teraction, e.g. Richard Bolt in the late seventies, today’sresearchers can benefit from various existing hardwaredevices and software toolkits. Although these develop-ment tools are available, using them is still a greatchallenge, particularly in terms of their usability andtheir appropriateness to the actual design and researchprocess. We present a three-part approach to support-ing interaction designers and researchers in designing,developing, and evaluating novel interaction modali-ties including multimodal interfaces. First, we presenta software architecture that enables the unification ofa great variety of very heterogeneous device driversand special-purpose toolkits in a common interactionlibrary named ”Squidy”. Second, we introduce a visualdesign environment that minimizes the threshold for itsusage (ease-of-use) but scales well with increasing com-plexity (ceiling) by combining the concepts of seman-tic zooming with visual dataflow programming. Third,we not only support the interactive design and rapidprototyping of multimodal interfaces but also provideadvanced development and debugging techniques to im-prove technical and conceptual solutions. In addition,

Werner A. Konig

E-mail: [email protected]

Roman RadleE-mail: [email protected]

Harald Reiterer

E-mail: [email protected]

Human-Computer Interaction Group

University of Konstanz, Box D-73http://hci.uni-konstanz.de

Tel.: +49 7531 88 3704

Fax: +49 7531 88 4772

we offer a test platform for controlled comparative eval-uation studies as well as standard logging and analysistechniques for informing the subsequent design itera-tion. Squidy therefore supports the entire developmentlifecycle of multimodal interaction design, in both in-dustry and research.

Keywords Multimodal User Interfaces · Post-WIMP user interface · natural interaction · designenvironment · zoomable user interface · semanticzooming · multimodal interaction · Squidy

1 Introduction

With recent advances in computer vision, signal pro-cessing, and sensor technology today’s researchers andinteraction designers have great opportunities to go farbeyond the traditional user interface concepts and in-put devices. More natural and expressive interactiontechniques, such as tangible user interfaces, interac-tive surfaces, digital augmented pens, speech input, andgestural interaction are available and technologicallyready to be incorporated into the multimodal interfaceof the future (see some examples in Figure 1). However,the actual utilization of these techniques for the designand development of multimodal interfaces entails var-ious critical challenges that interaction designers andresearchers have to face.

In contrast to the design of traditional graphicaluser interfaces, the development of multimodal inter-faces involves both software and hardware components[12]. However, conventional development environments(e.g. MS Visual Studio/.Net, Adobe Flash, Eclipse IDE)fall short of supporting uncommon interaction modali-ties and appropriate data processing (e.g. computer vi-sion), not to mention the handling of multipoint and

The original publication is available at http://www.springerlink.com

2

Fig. 1 Diverse input devices for single-modality or multimodal interfaces: (a) Physical game controller offer absolute pointing, motion

sensing and gesture-recognition to the end-user. (b) Digital pens build upon users’ pre-existing knowledge and thus offer a very naturalmode of interaction e.g. for digital sketching and prototyping. (c) Multi-touch surfaces augmented with physical tokens reduce the

gap between real-world and digital-world interaction. (d) Finger gestures provide a very expressive and direct mode of interaction.

(e) Well-known devices such as an omnipresent laser pointer provide flexible input from any distance.

Hardware platform Microsoft Surface Custom-build table Apple iPhone HTC Hero

Form factor table table mobile mobile

Operating system Microsoft Windows Linux/Windows Mac OS X Android OS

Programming language C# C++ Objective-C Java

Software framework Surface SDK Touchlib iPhone SDK Android SDK

Table 1 Interaction designers have to cope with very different environments for the same interaction modality, touch input.

multi-user applications (e.g. for multi-touch interaction).As a consequence a broad variety of very heterogeneousand specialized toolkits and frameworks have evolvedover the last few years such as Apple iPhone SDK1,Microsoft Surface SDK2, GlovePIE3, Processing4, NUIGroup Touchlib5. They provide support for specific in-teraction modalities, but are mostly restricted to a dedi-cated hardware environment and entail further require-ments and dependencies. When using touch as inputfor instance, the interaction designer has to cope withdifferent hardware platforms, operating systems, pro-gramming languages, and software frameworks (see Ta-ble 1). When developing single-modality interfaces, thisdiversity can be bypassed – at least in the short-run – byfocusing on just one specific device. But the combina-tion of multiple devices, e.g. for multimodal interactioninvolves further platforms, devices, and frameworks, re-

1 Apple iPhone SDK, http://developer.apple.com/iphone/2 Microsoft Surface SDK, http://www.microsoft.com/surface/3 GlovePIE, http://carl.kenner.googlepages.com/glovepie/4 Processing, http://processing.org/5 NUIGroup Touchlib, http://nuigroup.com/touchlib/

sulting in an unmanageable technical and mental com-plexity.

There are development environments that supportat least some of the more uncommon input devicesand modalities (e.g. physical turntables, mixing desks,multi-touch surfaces and simple vision tracking). Twoexamples are Max/MSP6 and vvvv7. Both are graphicaldevelopment environments for music and video synthe-sis and are widely used by artists to implement inter-active installations. Their popularity in the design andart community arises in particular from their graphi-cal user interface concepts. Both are based on the con-cept of visual dataflow programming and utilize a cable-patching metaphor to lower the implementation thresh-old [24] for interactive prototyping. Users arrange de-sired components spatially and route the dataflow be-tween the components by visually connecting pins in-stead of textual programming. However, the visual rep-resentation of each primitive variable, parameter, con-nection, and low-level instruction (e.g. matrix multipli-

6 Max/MSP, http://cycling74.com/products/maxmspjitter7 vvvv, http://vvvv.org/


3

cation) leads to complex and scattered user interfaces,even for small projects. vvvv offers the possibility of en-capsulating consecutive instructions in so-called ”sub-patches”. This approach helps to reduce the size of thevisual dataflow graph, but the hierarchical organizationintroduces additional complexity. In contrast to the vi-sual encapsulation in vvvv, the ”external” mechanismof Max/MSP supports the visual and technical encap-sulation of certain functionality in an external objectas a ”black-box”. This mechanism offers high flexibil-ity and abstraction but requires low level programmingin C. This results in a higher threshold and lower in-teractivity of the design and development process, sincechanges have to be textually written and compiled in anexternal development environment before the externalobject can be used in Max/MSP.

Basically, Max/MSP and vvvv show interesting userinterface concepts but they are focused on real-time au-dio composing and 3D rendering and were not designedto support the development of multimodal interfaces ingeneral. For that, interaction designers require not onlya set of ready-to-use interaction techniques and inputdevices but also the possibility to physically developand integrate new interaction modalities and hardwaredevices. Hardware toolkits such as Phidgets [11], Smart-Its [10] or iStuff [2] offer a set of compatible micro-controllers, sensor devices and software frameworks en-abling rapid prototyping of physical input and outputdevices. However, the technical complexity of the soft-ware frameworks requires advanced programming andsignal processing knowledge, in particular when multi-ple devices are used in parallel. iStuff mobile [1] com-bines the hardware toolkit iStuff with a visual program-ming environment based on Apple’s Quartz Composer8.This was originally designed to support the visual de-velopment of interactive multimedia and 3D render-ing. It shares the cable-patching metaphor with the al-ready discussed development environments vvvv andMax/MSP. This combination of hardware toolkit andvisual development environment facilitates fast itera-tions and rapid prototyping on multiple levels. How-ever, it is restricted to the domain of mobile phone in-teraction and limited in its functionality and the typeof input (e.g. no support for computer vision).

All of the aforementioned development environmentsand toolkits support diverse devices and modalities butthey are not especially designed to support the designof multimodal interfaces. Here, multiple inputs have tobe synchronized (e.g. hand-gesture and speech), pro-cessed and composed to a higher level command (e.g.moving an object). There are few frameworks that ad-

8 Apple Quartz Composer,

http://developer.apple.com/graphicsimaging/quartzcomposer/

dress these requirements. ICARE [5] is a conceptualcomponent model and a software toolkit for the rapiddevelopment of multimodal interfaces. It provides twotypes of software components: the elementary compo-nents, consisting of Device and Interaction Languagecomponents used to develop a specific modality, andthe Composition components that combine the diversemodalities. It was used for different use cases (e.g. de-sign of a multimodal flight cockpit) but it became ap-parent that only a limited set of the defined componentswere really generic [27] and the toolkit was not easilyextensible [22].

Based on the experiences gained with ICARE, theopen source framework ”OpenInterface” was developedby the OpenInterface Project9 that is dedicated to mul-timodal interaction. The OpenInterface framework iscomposed of the OpenInterface Kernel, a component-based runtime platform, and the OpenInterface Inter-action Development Environment (OIDE), a graphicaldevelopment environment for the design of multimodalinterfaces [27]. In order to integrate an existing inputdevice as component into the OpenInterface Kernel thecomponent interface has to be specified in an dedi-cated XML-based CIDL description language (Com-ponent Interface Description Language). This specifi-cation can be semi-automatically generated from thesource code of the component by the OpenInterfaceplatform. It also generates C++ code to encapsulatethe external binary into a well defined programminginterface [3]. Due to this explicit description of the in-terface the encapsulated component can be used in thegraphical development environment OIDE. This uses acable-patching metaphor similar to Max/MSP, vvvv,and Quartz Composer in order to define the dataflowby combining the selected components visually. Lawsonet al. [22] identified diverse shortcomings of the Open-Interface OIDE and the introduced application designprocess. A major issue is the limited focus and inflex-ible design of the components. The developers ratherfocus on the design of their individual component thanon the application as a whole. This leads to an in-flexible design of the components and the applicationin general that hinders the reuse, extension and ex-change of components as well as the entire applica-tion. This inflexibility also restricts interaction design-ers in exploring diverse alternatives, which then im-pedes rapid prototyping and limits epistemic produc-tion [18] of concrete prototypes. In order to address theidentified issues Lawson et al. introduced an all-in-oneprototyping workbench for multimodal application de-velopment [22]. It is a combination of the OpenInterfaceKernel with an Eclipse Plugin as graphical editor that

9 OpenInterface Project, http://www.oi-project.org/


4

is named SKEMMI. The editor is also based on thecable-patching metaphor, but provides three levels ofdetail with respect to the displayed information. Thelow-detail ”workflow” level reduces information and fa-cilitates the initial sketching of the desired interactiontechniques. In the ”dataflow” level where all details forthe routing of the dataflow are represented, the user se-lects, arranges and logically links the components with-out the need to route and connect every single pin.In a third-level, the ”component” level, only a specificcomponent with its input and output pins is visual-ized and the user is able to tailor the component’s in-terface (e.g. changing the port attributes and parame-ters). SKEMMI provides also an alternative source codeeditor view that allows for changes of the componentor its interface programmatically. The three-layer ap-proach helps to control the visual and functional com-plexity of the components, but there is no higher-levelabstraction concept (e.g. hierarchical pipelines or se-mantic zooming). If the designed multimodal interfaceincorporates multiple devices and various signal pro-cessing components, the SKEMMI user interface getsincreasingly crowded. The geometric zoom of the userinterface is not a solution for the complexity issue sinceit just changes the size of the displayed information butnot the information representation itself.

To sum up, there are only very few frameworksthat support the design of multimodal interfaces. How-ever, they either provide a limited range of interac-tion modalities or are hardly extensible regarding theplatform, the components or the visual user interface.The OIDE or the SKEMMI graphical editors seem verypromising, but the complexity issue is critical in realworld projects. Moreover, all of the discussed devel-opment environments focus mainly on rapid prototyp-ing and the early steps of iterative design. None ofthem provide tool-support for the empirical evaluationof the designed interfaces (e.g. ISO 9241-9 tapping tasksand suitable data-logging). All of the graphical develop-ment environments utilize the cable-patching metaphorin a similar way in order to connect input and out-put pins. However, the dataflow programming could bemore powerful without losing its simplicity. Further-more, they still require a deep understanding of theunderlying technology on behalf of the designers, sincethey have to understand and route each primitive vari-able/data item even when using ”black-box” modules.

In the following, we present our Squidy InteractionLibrary, which contributes on different levels:

– The software architecture: Squidy enables the uni-fication of heterogeneous devices and componentsin a common library. The architecture is designed

to provide great flexibility, simple extension, highindependency and fast parallel processing.

– The visual development environment: Squidy en-ables the interactive design and configuration of mul-timodal interfaces for interaction designers and re-searchers. The user interface concept is designed toprovide a low threshold (ease-of-learn) and high ceil-ing (high functionality) and scales well with increas-ing complexity.

– Tool-support for the entire development lifecycle:Besides the visual design and configuration for rapidprototyping, Squidy also provides advanced devel-opment and evaluation techniques for iterative de-sign.

After giving a short conceptual overview in the next sec-tion, we will discuss the software architecture in section2.1 and afterwards describe the user interface conceptin detail in section 2.2. In section 3 we will show theappropriateness of our solution to the actual design andresearch process in the context of a variety of real worldprojects.

2 Squidy Interaction Library

We introduce the Squidy Interaction Library, whichunifies a great variety of device toolkits and frameworksin a common library and provides an integrated user in-terface for visual dataflow management as well as deviceand data-filter configuration. Squidy thereby hides thecomplexity of the technical implementation from theuser by providing a simple visual language and a col-lection of ready-to-use devices, filters and interactiontechniques. This facilitates rapid prototyping and fastiterations for the design and development. However, ifmore functionality and profound customizations are re-quired, the visual user interface reveals more detailedinformation and advanced operations on demand by us-ing the concept of semantic zooming. Thus, users areable to adjust the complexity of the visual user interfaceto their current needs and knowledge (ease of learning).

The basic concept (see 2.2 for a more detailed dis-cussion) that enables the visual definition of the data-flow between the input and output is based on a pipe-and-filter concept (see Figure 2). By using this con-cept Squidy provides a very simple, yet powerful visuallanguage for designing the interaction logic. Users canselect an input device of choice as source, e.g. a laserpointer, which is represented by an input node in thevisual user interface. They connect it successively withfilter nodes for data processing, such as compensationfor hand tremor or gesture recognition and route therefined data to an output node as sink. Basically, the


5

Fig. 2 View of a simple pipeline in Squidy. The pipeline receives position, button and inertial data from a laser pointer, applies a

Kalman filter, a filter for change recognition and a filter for selection improvement and finally emulates a standard mouse for interacting

with conventional applications. At the same time the data is sent via TUIO to listening applications. The pipeline-specific functionsand breadcrumb navigation are positioned on top. The zoomable knowledge base, with a selection of recommended input devices,

filters, and output devices, is located at the bottom.

user defines the logic of an interaction technique bychoosing the desired nodes from a collection (knowledgebase) and connecting them in an appropriate order as-sisted by a heuristic-based node suggestion. The filternodes are independent components that can transmit,change, or delete data objects, and also generate addi-tional ones (e.g. if a gesture is recognized). The sourceand sink are specific drivers that handle the input /output operations and map the individual data formatof the devices to the generalized data types defined inSquidy (see Figure 4). The pipe-and-filter concept pro-vides also very technical advantages, since the encap-sulation of functionality in independent ”black-boxes”ensures information hiding, modifiability and high reuseby abstraction. The possibility for multiple input andoutput connections offers a high degree of flexibility andthe potential for massive parallel execution of concur-rent nodes. In our implementation each node generatesits own thread and processes its data independently assoon as it arrives. This effectively reduces the processingdelay that could have a negative effect on the interac-tion performance.

The sink can be any output technique such as a vi-brating motor for tactile stimulation or LEDs for vi-sual feedback. Squidy also provides a mouse emula-tor as an output node to offer the possibility of con-trolling standard WIMP-applications with unconven-tional input devices. Multipoint applications (e.g. formulti-touch surfaces or multi-user environments) andremote connections between multiple Squidy instancesare supported by an input / output node that trans-mits the interaction data either as TUIO messages [17]or as basic OSC messages over the network. TUIO is awidely used protocol for multipoint interaction based onthe more general OpenSound Control protocol (OSC),which is a successor to the MIDI standard. By provid-ing these standard interfaces for both input and out-put connections Squidy supports the majority of multi-touch applications that have recently become very pop-ular in both research and industry. Above these basicnetwork interfaces Squidy also supports and integratesmore complex frameworks such as the Apple iPhoneSDK, the Android SDK, the NUIGroup Touchlib, andthe Microsoft Surface SDK. Users therefore benefit from


6

Fig. 3 Input node in Squidy representing an interactive laserpointer. In order to reduce visual complexity the node-specific

functions (active/inactive, delete, duplicate, publish to knowledge

base) and the unconnected in and out ports are only shown if thepointer is within the node.

the particular functionalities and specific hardware ofall these techniques. Inside Squidy, however, they arealso able to define, control, evaluate, and reuse interac-tion techniques independently from the hardware or thespecific framework. This flexibility results from the ar-chitecture utilized and the generalized data types whichwill be explained in more detail in the following section.

2.1 Software Architecture

There are several frameworks and toolkits that provideready-to-use components for input devices and signalprocessing. Instead of connecting the components topipelines programmatically, most of these frameworksand toolkits offer a basic language for controlling thedataflow visually (for example Max/MSP, vvvv, OIDEor SKEMMI). Such a visual programming language re-duces the technical threshold and complexity and aidsusers with little or no programming experience. Also,the integration of new modalities requires a fine graspof the underlying technology and thus is still a highlydemanding task. Although, extending a framework withnew components is only offered by a few of today’scommon frameworks such as ICARE [4] or the opensource framework OpenInterface (www.oi-project.org).However, integrating new components into the frame-works requires either an additional programming effortor a dedicated definition of the interface by a specificmark-up language. Basically this means that a devel-oper has to switch between different applications andprogramming languages while developing a new inter-action technique, increasing the mental workload.

2.1.1 Generic Data Types

In order to unify very heterogeneous devices, toolkitsand frameworks, we generalized the various kinds ofinput and output data to a hierarchy of well-defined

Fig. 4 Data type hierarchy in Squidy based on primitive virtual

devices [30]. Any data processed in Squidy consists of single or

combined instances of these basic data types.

generic data types (see Figure 4) based on the primitivevirtual devices introduced by Wallace [30] and adaptedto the work of Buxton [6] and Card et al. [7]. Eachgeneric data type consists of a type-specific aggrega-tion of atomic data types such as numbers, strings orBoolean values bundled by their semantic dependency.Simply adding a single connection between two nodesin the visual user interface performs routing of dataflowbased on these generic data types.

This is quite a different approach when comparedto some of the aforementioned frameworks such as theICARE [5] and vvvv. These frameworks use atomicdata types defined in the particular programming lan-guage and assign them visually by connecting resultvalues with function arguments in their specific userinterfaces. In order to use the functionality of a mod-ule in these frameworks, the user has to route each ofthese low-level data types. Each x-, y-, and z-value ofa three-dimensional data type has to be routed sepa-rately, for example. This is a procedure that needs addi-tional effort and can be error-prone, in particular whendesigning complex interaction techniques. Furthermore,this approach requires detailed knowledge about thefunctionality of each node and its arguments. Routinglow-level data types therefore puts high cognitive loadon the user and leads to visually scattered user inter-faces, particularly as the number of connected nodesincreases.

Squidy, on the other hand, does not require the de-signer to visually define every value and programmingstep manually. The interaction data is grouped in se-mantically bundled generic data types as mentionedbefore. Squidy therefore offers the abstraction and sim-plicity of a higher-level dataflow management and re-duces the complexity for the interaction designer with-out limiting the required functionality.


7

Fig. 5 This figure shows the usage scenario of an interactive and multimodal environment to control an application running on a

360◦ panorama screen by using touch gestures and speech. The user interacts with his fingers by touching the display of an Apple

iPhone (1). All recognized touches will be sent from an iPhone Client application (OSC reference implementation running on theiPhone) to the OSC Bridge of Squidy (2). The Squidy Core will process the incoming data appropriately and sent it via the “special

purpose bridge” (3) to the 360◦ application (4) to control a cursor object, which visually highlights the users current finger position.If the user has selected an interactive element with such a touching gesture the application (5) sends a tactile feedback back to its

connected bridge (6). The tactile feedback coming from the application will be forwarded through the OSC Bridge (7) to the iPhone

(8) where the vibration motor will be activated to inform the user that he is hovering above an interactive element. After the user hasrealized the tactile feedback and thus the interactive element (9), he will use a spoken command to invoke an action on the selected

object. Therefore, the spoken command will be recognized by the operating system’s speech recognition and then will be sent to the

“native interface bridge” (10). The appropriate spoken command will have been processed by the Squidy Core (11) and transformedinto an action, which will be sent to the application to trigger object activation / manipulation (12). This multimodal scenario can be

implemented with Squidy using pluggable Squidy Bridges to receive data from different devices and a simple arrangement of nodes to

process that incoming data.

2.1.2 Squidy Bridge

In order to achieve high extensibility and to simplifythe integration of new devices and applications, we pro-vide the Squidy Bridges as common interfaces that sup-port widely used network protocols and also offer aspecific native API if high-performance data transmis-sion is needed. For the purpose of unifying data pro-duced by different hardware devices or applications (es-pecially relevant for incorporating multiple interactionmodalities), the Squidy Bridges map the diverse data

originating from heterogeneous sources into the genericdata types. Thus, the internal data processing is har-monized and completely separated from the diversityof the external world. These bridges are able to handledata transformations in both directions (e.g. from Ap-ple iPhone into the Squidy Core and from the SquidyCore to the application running on the panoramic dis-play and vice versa in order to close the feedback loope.g. activation of the vibrator on the iPhone as tactilefeedback of the application’s status (see Figure 5)). Theinteraction library already comes with an OSC Bridge


8

Listing 1 OSC Message sent from an Apple iPhone contains

four arguments (finger touch)

/∗∗∗ 1 . g e n e r i c data type∗ 2 . IMEI as i d e n t i f i e r

∗ 3 . x−p o s i t i o n∗ 4 . y−p o s i t i o n∗/

String : de . ukn . hc i . squidy . core . data . Posit ion2D

String : 49 015420 323751 8double : 0 .25double : 0 .17

and a Native Interface Bridge that can be used out-of-the-box. The OSC Bridge offers the possibility ofdirectly connecting the various available devices andtoolkits using this communication protocol. Since OSCis based on standard network protocols such as UDPor TCP, it is highly flexible and widely applicable, inparticular for mobile or ubiquitous computing. An OSCmessage consists of several arguments such as the classname of the generic data type, a unique identifier anddata-type-specific parameters. For instance, a messagefor a two-dimensional position that may be sent from anApple iPhone would contain the Position2D data typeas first argument, IMEI number as second argument, x-and y-value as third and fourth argument (Listing 1).

The flexibility gained from this network approach(e.g. hardware and software independence, high scala-bility by distributed computing (see Figure 15)) entailsa certain delay that can have a negative effect on userinput performance [23]. Thus, for those purposes whenperformance is more important than flexibility, the Na-tive Interface Bridge provides a straightforward Javaand C/C++ API to map data from individual devicesto the generic data types in Squidy programmatically.In contrast to the OSC Bridge, this technique increasesthroughput and reduces the delay to a minimum.

For devices that support neither the OSC proto-col nor the Native Interface Bridge by default, Squidyprovides client reference implementations (e.g. SquidyClient for iPhone OS10 and for Android OS11) that canbe deployed on these devices, minimizing the effort andthreshold of device integration. However, if the hard-ware is not able to communicate via existing bridgesnatively, or if deployment of proprietary software is notdesired or is not possible due to hardware restrictions,then users can add further bridges to allow commu-

10 Squidy Client for iPhone OS:

http://itunes.apple.com/app/squidy-client/id32933592811 Squidy Client for Android OS:

http://sourceforge.net/projects/squidy-lib/

files/Components/Squidy-Client-for-Android-OS

nication, for instance through special-purpose proto-col bridges such as the Virtual-Reality Peripheral Net-work [29].

The support of multiple bridges as interfaces in com-bination with the device-independent generic data typesenables a separation of the data sources and the sig-nal processing in the Squidy Core. This offers a simplebut flexible integration of new interaction techniquesand modalities without touching existing core function-ality. As with ICARE [4] or OpenInterface (www.oi-project.org), interaction techniques designed with theuser interface are completely decoupled from the indi-vidual hardware or the connected applications. Replac-ing devices (e.g. switching from the Apple iPhone to theMicrosoft Surface) therefore does not affect the appliedinteraction techniques (e.g. ”selection by dwelling”) orthe concrete application also connected to a SquidyBridge. The independent-bridge approach in combina-tion with the generalization of data types enables theintegration of very heterogeneous devices and toolkitsin Squidy. Interaction techniques that have been de-fined once can be reused multiple times. Squidy thusreduces complexity by abstraction, offers high flexibil-ity and enables rapid prototyping.

2.1.3 Squidy Core

All data resulting from (multimodal) user interactionis bridged from devices to the Squidy Core. The coreprocesses this data automatically and in parallel with-out any programming effort or further customizations.Users can define a filter chain (processing chain) us-ing visual dataflow programming provided by the vi-sual user interface of the Squidy Interaction Library.In order to process the interaction data, the SquidyCore provides a flexible API for manipulating (CRUD– Create / Read / Update / Delete) the dataflow. Toinsert new or changed data objects into the dataflow,the publish-method (Listing 2) of the API can be calledat the desired place in the pipeline. For instance, a ges-ture recognizer that has detected a pre-defined gesturewill publish a new gesture object into the dataflow.These methods accept 1...n instances of data objectsor a data container that consists of an array of dataobjects as well as a release timestamp. The interface’IData’ ensures the compatibility of the published dataobjects with the generic data types defined in Squidyand specifies common methods and enumerations.

Furthermore, the Squidy Interaction Library comeswith diverse off-the-shelf filters for signal processing,data fusion, filtering and synchronization that providethe essential functionalities for developing multimodalinterfaces. Compared to OIDE [27] or SKEMMI [22],


9

Listing 2 Methods to insert new or changed data objects into

the dataflow.

/∗∗∗ Publ i shes 1 . . . n data o b j e c t s to enhance the∗ dataf low semant ics .

∗/public void pub l i sh ( IData . . . data ) ;

/∗∗∗ Publ i shes a data conta ine r that c o n s i s t s o f∗ an array o f data o b j e c t s and a timestamp on∗ which the data conta ine r has been r e l e a s e d .

∗/public void pub l i sh ( IDataContainer

dataContainer ) ;

Squidy incorporates the facility to add new filters (in-cluding properties, algorithms, logic and descriptions)without the need for switching to a different develop-ment environment. Therefore, the source code is embed-ded and can be manipulated by users directly. Changesmade to the source code will be compiled and inte-grated on-the-fly and the new or changed functional-ity is thus instantly available to users. Each implemen-tation of a filter owns a data queue and a processingthread without any effort on the developer’s part. Theincoming data will be enqueued until the processingthread dequeues data to perform custom data process-ing automatically [5]. Thus, the interaction library runsin a multi-threaded environment that allows concurrentdata processing by each filter without blocking the com-plete process chain (e.g. a filter that is currently wait-ing for a system resource does not block other filtersduring that time). This system of self-contained filtercomponents prevents side effects on the signal process-ing and thus aids users to design consistent and reliableinteraction techniques. Users can intercept a filter’s in-ternal processing by implementing simple pre-definedmethod stubs similar to the concept of ”Method CallInterception”. The following method stubs reflect dif-ferent points of entry that differ in the quantity andtype of dequeued data provided. The processing threaddetermines in a certain sequence whether a stub is im-plemented and then invokes this stub using reflection.

In the ”preProcess” stub (Listing 3), the collectionsof data types grouped within a data container are passedto the method’s implementation. This is an easy way toaccess all data at a glance or iterate through the datacollection manually, e.g. to search for interaction pat-terns consisting of a diverse set of data types concern-ing multimodal interaction. Whenever it is sufficient toprocess one particular data instance at a time, the ’pro-cess’ method stub is appropriate. The code fragment in

Listing 3 The “preProcess” stub grants access to all data of a

data container.

/∗∗∗ Diverse c o l l e c t i o n o f data a c c e s s i b l e by∗ t h i s method stub be f o r e i n d i v i d u a l

∗ p r o c e s s i n g .∗/

public IDataContainer preProces s (IDataContainer dataContainer ) ;

Listing 4 Processing single data objects of a specified type at atime.

/∗∗∗ Proce s s e s data o f p a r t i c u l a r g e n e r i c data∗ type (DATA TYPE i s a p l a c eho lde r f o r

∗ those g e n e r i c data types )∗/

public IData proce s s (DATA TYPE data ) ;

Listing 5 All data objects of a data container are accessible

through the “postProcess” stub after individual data processing.

/∗∗∗ Diverse c o l l e c t i o n o f data a c c e s s i b l e by

∗ t h i s method stub a f t e r i n d i v i d u a l∗ p r o c e s s i n g .∗/

public IDataContainer pos tProces s (IDataContainer dataContainer ) ;

Listing 4 is a generic representation of such a processmethod stub.

In the case of the “process” stub (Listing 4), theSquidy Core iterates through the collection automati-cally. It therefore does not have to be done programmat-ically as in the “preProcess” stub. Here, DATA TYPEis the placeholder for a generic data type (section 2.1.1),offering a simple data-type filter for the dataflow. TheSquidy Core only passes instances of that generic typeto that method implementation.

Before the data collection is published to the nextfilter of the processing chain or bridged back to any de-vice or application, the data collection can be accessedthrough the ”postProcess” stub (Listing 5). An exam-ple of using this post processing is the functionalityto remove duplicate data from the dataflow to reducedata-processing overhead.

The Squidy Core uses the Java Reflection mecha-nism to determine if a filter has implemented such adata interception and passes inquired data to the im-plementation automatically. Therefore, no additionaleffort is required for interface declaration, generationand compilation such as is needed for the CIDL used


10

by the OpenInterface framework (www.oi-project.org).This flexibility of the Squidy Core to quickly integrateor modify filter techniques provides the capability oftenneeded to rapidly and iteratively prototype interactiveand multimodal interfaces.

Heterogeneous devices and toolkits can be easilytied to the Squidy Interaction Library using existingSquidy Bridges (OSC Bridge, Native Interface Bridge)or custom bridge implementations (e.g. to integrate de-vices or toolkits communicating via special protocols).The Squidy Core provides a multi-threaded environ-ment to perform concurrent data processing and thusincreases data throughput, minimizes lag and enhancesuser’s experience while using multimodal interaction.A suitable API supports developers to quickly imple-ment new filters or change existing filters without theneed for recompilation or repackaging. The three-tierarchitecture covers usage by both interaction designersand developers, assists them with appropriate tools andthus reduces mental activity to a minimum.

Currently we run applications based on Microsoft.Net, Windows Presentation Foundation and SurfaceSDK, Adobe Flash and Flex, OpenGL for C++ or JOGLas well as standard Java technology. The Squidy Bridgescombined with Squidy Client reference implementationsprovide various external and integrated drivers and toolk-its. Currently, Squidy supports the NUIGroup Touch-lib, the Apple iPhone SDK, the Android SDK and Mi-crosoft Surface SDK for multi-touch interaction, theART DTrack and the NaturalPoint OptiTrack for fin-ger gestures [9] and body-tracking, the libGaze for mo-bile eye-tracking [14], the iPaper framework for penand paper-based interaction [28], the Microsoft Touch-less SDK for mid-air object tracking, the Phidgets APIfor physical prototyping and self-developed componentsfor laser pointer interaction [19], GPU-accelerated low-latency multi-touch tracking (SquidyVision), NintendoWii Remote and tangible user interface (TUI) interac-tion.

2.2 User Interface Concept

The Squidy user interface concept is based on the con-cept of zoomable user interfaces. It is aimed at provid-ing different levels of details and integrating differentlevels of abstraction so that frequent switching of ap-plications can be avoided. In the following subsectionswe will provide more details about the different userinterface concepts.

Fig. 6 The Squidy Knowledge Base is a searchable interface for

accessing all implemented input device and filter nodes.

2.2.1 Knowledge Base

Squidy provides a wide range of ready-to-use devicesand filter nodes stored in an online knowledge base thatis accessible within the Squidy user interface. An assort-ment is directly offered at the bottom of the pipelineview (see Figure 2). The selection and arrangement ofthe nodes are based on statistics of previous usage andthus give a hint of suitable partners for the currentlyfocused device or filter. This dynamic suggestion maylead to a higher efficiency and also helps novice users tolimit the otherwise overwhelming number of availablenodes to a relevant subset. The user can directly drag adesired node from the selection (bottom) to the designspace for the pipeline (centre). If the desired node is notpart of the suggested subset, the user has the possibilityof accessing all nodes of the knowledge base by zoom-ing into the corresponding view at the bottom. Therein,dynamic queries support the exploration (see Figure 6).These are based both on automatically generated meta-data about each node as well as user-generated tags.

2.2.2 Semantic Zooming

In accordance with the assumption that navigation ininformation spaces is best supported by tapping intoour natural spatial and geographic ways of thinking[25], we use a zoomable user-interface concept to navi-gate inside the Squidy visual user interface. When zoom-ing into a node, additional information and correspond-ing functionalities appear, depending on the screen spaceavailable (semantic zooming). Thus, the user is able togradually define the level of detail (complexity) accord-ing to the current need for information and functional-ity.


11

Fig. 7 View of a zoomed Kalman filter node with table of pa-

rameters. Parameter changes are applied immediately. Spatialscrolling with overview window (right) and temporal scrolling

of last changes (bottom) is provided visually. Via automatic

zooming, the user can access further information about the node(Fig. 8) and the filter source code (Fig. 9).

2.2.3 Interactive Configuration & Evaluation

In contrast to the related work, the user does not haveto leave the visual interface and switch to additionalapplications and programming environments in orderto get additional information, to change properties, orto generate, change or just access the source code of de-vice drivers and filters. In Squidy, zooming into a nodereveals all parameters and enables the user to inter-actively adjust the values at run-time (see Figure 7).The changes take place immediately without any needfor a restart, providing a direct relationship betweenuser interaction and application feedback and therebymaintaining causality, as Card et al. puts it [8]. This isespecially beneficial for empirically testing a number ofdifferent parameters (e.g. adjusting the noise levels ofa Kalman filter) because of the possibility of directlycomparing these settings without introducing any (e.g.temporal) side effects. This process of interactive con-figuration and evaluation is much needed during thedesign of multimodal interaction, especially when usinguncommon interaction techniques and user interfaces.Squidy therefore facilitates fast development iterations.

2.2.4 Details on demand

Going beyond the access and manipulation of parame-ters, Squidy provides illustrated information about thefunctionality, usage and context of the node, and thisinformation is directly embedded in the node. By zoom-ing into the information view marked by a white ”i”on a blue background (see Figure 7), the information

Fig. 8 Information view of the Kalman filter node providing

illustrated descriptions about its functionality.

Fig. 9 Source Code of the corresponding device or filter node

is directly accessible by semantic zooming. Zooming-out leads toruntime compilation of the source code and live integration into

the current pipeline.

is shown without losing the context of the node. Thisinformation view (see Figure 8) may contain code doc-umentation (e.g. automatically generated by javadoc),user-generated content (e.g. from online resources suchas wikipedia.org or the Squidy-Wiki) or specifically as-sembled documentation such as a product specificationconsisting of textual descriptions, images or videos. Theinteraction designer using Squidy does not need to opena web browser and has to search for online documenta-tions in order to get the relevant information. Due tothe semantic zooming concept the user specifies her in-formation need implicitly by navigating in the zoomableuser interface and spatially filtering the informationspace.


12

2.2.5 Embedded Code and on-the-fly compilation

The user even has the ability to access the source code(see Figure 9) of the node by semantic zooming. Thus,code changes can be made directly inside the designenvironment. Assistants such as syntax highlighting orcode completion support the user even further. If theuser zooms out, the code will be compiled and inte-grated on the fly, again without needing to restart thesystem. Users may also generate new input and outputdevices or filters by adding an empty node and aug-menting it with applicable code. In order to minimizethe threshold for the first steps and to reduce the writ-ing effort, the empty node already contains all relevantmethod definitions for data handling and processing.Therefore, only the desired algorithm has to be filled inthe suitable method body of the node. By zooming outthe new node is compiled and it is than immediatelyready for usage. In order to share the new node withthe community the user can publish it into the knowl-edge base (see Publish-button in Figure 3 and 2). Thedesign rationale is not to replace the classical develop-ment environments such as Microsoft Visual Studio orEclipse, but rather to integrate some of their function-ality directly into Squidy. Thereby, we provide a unifiedenvironment that seamlessly integrates the most rele-vant tools and functionalities for the visual design andinteractive development of multimodal interfaces.

2.2.6 Dataflow Visualization - Visual Debugging

The visual design of an interaction technique requiresa profound understanding of the data flow and the se-mantics of the designed pipeline. For instance, to detectand analyze interaction patterns such as gestures ormultimodal input, researchers or interaction designersshould be able to quickly get an overview of the inter-action data flow during a particular time span. In com-pliance with the pipe-and-filter metaphor, we integratea data-flow visualization at the centre of each pipe (seeFigure 2). This simple yet powerful view (see Figure 10)visualizes the data flow through its corresponding pipewith respect to its temporal and spatial attributes. Ata glance, users are able to inspect a massive amount ofdata, as well as data occurring in parallel, according toits spatial location and chronological order.

Direct manipulation of the time span allows the userto adjust the range to their current need. The visualrepresentation of data depends on the type that thedata belongs to (e.g. representation of a position in 2Ddiffers from the representation of a gesture being rec-ognized - see Figure 10). Thus, users benefit from theinsight into the interaction data flow by getting a bet-

Fig. 10 Dataflow visualization showing the values of all for-

warded data objects of a pipe within a defined time span.

ter understanding of the effect of different parametersettings.

Every node in Squidy operates strictly within itsown thread and therefore implies multi-threading andconcurrent data processing without any additional ef-fort. This allows a higher bandwidth and enhances thedata throughput. Nevertheless, users may produce er-rors while implementing nodes or use incorrect parame-ter settings. This can cause side effects (e.g. array indexout of bounds) that in consequence may lead to an in-accurate interaction technique or a local error (othernodes run in separate threads and are therefore un-affected). Thus, the design environment supplies eachproject, pipeline, node and pipe (in the following wecall these shapes) with a visual colour-coded outer-glow effect (see Figure 2) that represents the node’scurrent status. Three distinct colours (green, red, grey)are uniquely mapped to a class of conditions. A greenglowing shape indicates a properly operating node im-plementation running underneath. Additionally, pipespossess a green illumination when interaction data isflowing or has recently been flowing through them. Thered glow indicates that an error has occurred duringexecution of node implementation (e.g. unhandled ex-ception - NullPointerException). Then, all connectedoutgoing pipes to a defective pipeline or node are giventhe same error colour-coding status to enhance errordetection and allow faster error correction. Shapes thatare not set as activated (not running) and pipes thatcurrently do not have interaction data flowing throughreceive a grey illumination. Thereby, without any needfor interaction, the user can perceive the status of thedata flow between the nodes of the pipeline.

Occasionally, researchers or interaction designers re-quire the capability to preclude parts of interaction data


13

Fig. 11 Globorama installation on a 360◦ panoramic screen. On the left: top view of the panorama screen with 8m in diameter. On

the right: Visitors exploring the Globorama installation with an interactive laser pointer.

from being propagated to nodes (e.g. disabling the but-tons pressed on a laser pointer and instead using ges-tures to control an application). Thus, a pipe providestwo opportunities to narrow the set of interaction dataflowing through it. The first possibility for reducing theset of interaction data is before data is visualized bythe dataflow visualization. This allows the user to vi-sually debug designated types of data. The second pos-sibility is immediately after the data comes out of thedataflow visualization. The user can visually debug thedata but nevertheless prevent it from being forwardedto nodes connected downstream. Users are able to zoominto the data-type hierarchy view (see Figure 4) and se-lect (which means this data is forwarded) or deselect adata type by clicking on it. In Figure 4 all data types areselected and therefore have a coloured background. Adeselected data type would just have a coloured border.

3 Squidy Use Cases

Over the last two years, we iteratively developed, usedand enhanced Squidy during the course of applying itin several diverse projects. The starting point was theneed for an infrastructure that facilitates the design andthe evaluation of novel input devices and interactiontechniques in multimodal and ubiquitous environments.

The first input device that we implemented withSquidy was an interactive laser pointer. This enabled aflexible interaction with a large, high-resolution displaysuch as the Powerwall located at the University of Kon-stanz (221 inches, 8.9 megapixels) from any point anddistance [19]. A major issue of this interaction techniquewas the pointing imprecision introduced by the naturalhand tremor of the user and the limited human hand-eye coordination [21]. Squidy improved the design and

research process by providing the opportunity to inter-actively implement, change and empirically test diversesmoothing techniques without introducing side effects.In an iterative approach the dataflow was visualized,the filter logic was adapted, the filter parameters wereoptimized, and the resulting interaction technique wasfinally evaluated based on a Fitts’ Law Tapping Test(ISO 9241-9), which is also provided in Squidy as aready-to-use component (see Figure 12). Thus, Squidysupported the entire development lifecycle, resulting ina very efficient and effective project progress.

Fig. 12 Laser Pointer Interaction in front of a large high-resolution display. Squidy facilitated the integration and evalua-

tion of precision enhancing and smoothing techniques, allowingthe precise selection of targets as small as 22mm in diameter from

a 3m distance [21].

In a follow-up project we specifically made use of theseparation of the three layers in Squidy since we couldeasily apply the laser pointer interaction to an artisticinstallation. This scenario utilized the laser pointer forinteraction but came with a very different display andvisualization technique. Surrounded by 360◦-satelliteimages of the earth, visitors to the ”Globorama” instal-lation explored the entire globe with the laser pointer


14

and submerged at selected points in geo-referenced pano-ramic photographs or webcam images of the respec-tive location (see Figure 11). The visitors were ableto physically move inside the 360◦ panoramic screen(8m in diameter, 8192x928px) while dynamically con-trolling zooming and panning and selecting interestingareas [20]. We also augmented the laser pointer with avibrator for tactile feedback and multiple RGB-LEDsfor visual feedback on the device. The tactile feedbackwas given whenever the visitor moved the cursor overan active element such as a geo-referenced photographand the color LEDs visualized the current status of thesystem. The ”Globorama” installation was exhibited atthe ZKM Center for Art and Media in Karlsruhe (2007)and at the ThyssenKrupp Ideenpark 2008 in Stuttgart.

Fig. 13 The Media Room is a lab environment which provides a

variety of input and output devices with different modalities andform factors. Squidy serves as the basic technology to integrate

these different devices as well as to configure and evaluate them.

Squidy was also used to design functional prototypesfor personal information management with interactivetelevision sets [15]. For this application domain, theNintendo Wii, in its role as a standard input devicefor home entertainment, was integrated into Squidy.Although the device, the application, and the displaywere completely different to the previous projects, thesmoothing filters implemented for the laser pointer couldbe applied to the Wii and proved to be very benefi-cial, since both the Nintendo Wii and the laser pointershare an important similarity in being absolute point-ing devices. Furthermore, the wiigee gesture recogni-tion toolkit [26] was integrated into Squidy to enablethree-dimensional gestures with the Nintendo Wii. Al-though the toolkit was originally designed for the Nin-tendo Wii, the laser pointer can be used interchange-ably, since Squidy unifies the individual data types ofthe devices with the generic data types commonly de-fined in Squidy.

Fig. 14 Multi-touch surfaces augmented with physical tokens

used in the context of blended museum.

In the context of Surface Computing, we conceptu-ally and technically combined multiple touch-sensitivedisplays aiming to provide a more ubiquitous user expe-rience based on the naturalness and directness of touchinteraction [16]. In this scenario, we integrated mobilehandhelds (Apple iPhone) as personal devices as well asshared multi-touch tables (Microsoft Surface) and largehigh-resolution walls (eyevis Cubes) for collaborativedesign and visual information organization. In orderto facilitate multimodal input and context-aware appli-cations, we integrated speech recognition, mobile eyetracking [14] and freehand gestures [9]. To further closethe gap between the digital and the physical world, weenhanced this environment with digital pens for inter-active sketching and the possibility of interacting withphysical tokens on the diverse multi-touch displays (seeFigure 14). All of these techniques were integrated in,and driven by, Squidy and installed in our interactionlab known as the Media Room (see Figure 13). Thisphysical infrastructure in combination with Squidy asthe common software infrastructure gives an ideal de-sign and development environment for researchers andinteraction designers developing the user interfaces ofthe future.

4 Conclusion and Future Work

”Creating interactive systems is not simply the activ-ity of translating a pre-existing specification into code;there is significant value in the epistemic experience ofexploring alternatives”, (Hartmann et al. [13]).

This statement is especially true for the design ofmultimodal interfaces, since there is no well establishedbody of knowledge and no ready-to-use solution formultimodal interfaces the designer can take advantage


15

Fig. 15 This cloud shows how Squidy contributes to the development lifecycle of multimodal interaction techniques.Each phase in the lifecycle, whether it is the design and prototyping, the implementation and testing, or the usability

evaluation phase is surrounded by a variety of Squidy features that support the interaction designer or developerduring this activity.

of. Interaction designers need to physically explore andprototype new interaction modalities and therefore re-quire development environments that especially sup-port the interactivity and the dynamic of this creativedevelopment process. We presented the Squidy Inter-action Library that supports the interactive design ofmultimodal user interfaces with a three-part contribu-tion. First, it provides a software architecture that of-fers the flexibility needed for rapid prototyping and thepossibility to integrate a vast variety of heterogeneousinput devices and signal processing filters. Second, theSquidy visual user interface introduces a new user in-terface concept that combines visual dataflow program-ming with semantic zooming in order to reduce the vi-sual and technical complexity. This visual approach en-ables also a high degree of interactivity that is furthersupported by the fluid integration of code views, filtermechanisms and visualization tools. Third, the SquidyInteraction Library does not only focus on the rapidprototyping, but also provides advanced developmenttechniques and tool-support for empirical evaluation ofthe developed interfaces. Figure 15 shows a high-levelfeature cloud of the Squidy Interaction Library withrespect to the different development phases. The ap-propriateness of Squidy to the actual design and re-

search process was practically shown by the presenteduse cases. Additionally, we will conduct qualitative us-ability tests in order to validate and inform the designof the Squidy user interface concept. Up to now, theSquidy Interaction Library has not provided multi-usersupport. This, and the integration of version control-ling, will be future work.

The Squidy Interaction Library is free software andpublished at http://hci.uni-konstanz.de/squidy/under the licence of the LGPL.

References

1. Rafael Ballagas, Faraz Memon, Rene Reiners, and JanBorchers. istuff mobile: rapidly prototyping new mobilephone interfaces for ubiquitous computing. In CHI ’07: Pro-ceedings of the SIGCHI conference on Human factors in

computing systems, pages 1107–1116, New York, NY, USA,2007. ACM.

2. Rafael Ballagas, Meredith Ringel, Maureen Stone, and JanBorchers. istuff: a physical user interface toolkit for ubiq-

uitous computing environments. In CHI ’03: Proceedings ofthe SIGCHI conference on Human factors in computing sys-

tems, pages 537–544, New York, NY, USA, 2003. ACM.

3. Alexandre Benoit, Laurent Bonnaud, Alice Caplier, Frederic

Jourde, Laurence Nigay, Marcos Serrano, Ioannis Damousis,


16

Dimitrios Tzovaras, and Jean-Yves Lionel Lawson. Multi-modal signal processing and interaction for a driving simula-

tor: Component-based architecture. Journal on Multimodal

User Interfaces, 1(1):49–58, 2007.

4. Jullien Bouchet and Laurence Nigay. Icare: a component-based approach for the design and development of multi-

modal interfaces. In CHI ’04: CHI ’04 extended abstractson Human factors in computing systems, pages 1325–1328,

New York, NY, USA, 2004. ACM.

5. Jullien Bouchet, Laurence Nigay, and Thierry Ganille. Icare

software components for rapidly developing multimodal in-terfaces. In ICMI ’04: Proceedings of the 6th international

conference on Multimodal interfaces, pages 251–258, New

York, NY, USA, 2004. ACM.

6. William Buxton. Lexical and pragmatic considerations ofinput structures. SIGGRAPH Comput. Graph., 17(1):31–

37, 1983.

7. Stuart K. Card, Jock D. Mackinlay, and George G. Robert-

son. A morphological analysis of the design space of inputdevices. ACM Trans. Inf. Syst., 9(2):99–122, 1991.

8. Stuart K. Card, Allen Newell, and Thomas P. Moran. The

Psychology of Human-Computer Interaction. L. ErlbaumAssociates Inc., Hillsdale, NJ, USA, 1983.

9. Stephanie Foehrenbach, Werner A. Konig, Jens Gerken, and

Harald Reiterer. Natural interaction with hand gestures and

tactile feedback for large, high-res displays. In MITH 08:Workshop on Multimodal Interaction Through Haptic Feed-

back, held in conjunction with AVI 08: International Work-

ing Conference on Advanced Visual Interfaces, 2008.

10. Hans Gellersen, Gerd Kortuem, Albrecht Schmidt, andMichael Beigl. Physical prototyping with smart-its. IEEE

Pervasive Computing, 3(3):74–82, 2004.

11. Saul Greenberg and Chester Fitchett. Phidgets: easy devel-

opment of physical interfaces through physical widgets. InUIST ’01: Proceedings of the 14th annual ACM symposium

on User interface software and technology, pages 209–218,

New York, NY, USA, 2001. ACM.

12. R. Harper, T. Rodden, Y. Rogers, and A. Sellen. Being Hu-man: Human-Computer Interaction in the Year 2020. Mi-

crosoft Research Ltd., Cambridge, 2008.

13. Bjorn Hartmann, Leith Abdulla, Manas Mittal, and Scott R.Klemmer. Authoring sensor-based interactions by demon-

stration with direct manipulation and pattern recognition. In

CHI ’07: Proceedings of the SIGCHI conference on Humanfactors in computing systems, pages 145–154, New York, NY,

USA, 2007. ACM.

14. S. Herholz, L.L. Chuang, T.G. Tanner, H.H. Blthoff, and

R. Fleming. Libgaze: Real-time gaze-tracking of freely mov-ing observers for wall-sized displays. In VMV ’08: Proceed-

ings of the 13th International Workshop on Vision, Model-

ing, and Visualization, 2008.

15. Hans-Christian Jetter, Andreas Engl, Soeren Schubert, andHarald Reiterer. Zooming not zapping: Demonstrating thezoil user interface paradigm for itv applications. In Ad-

junct Proceedings of European Interactive TV Conference,Salzburg, Austria, July 3-4, 2008, Jul 2008. Demonstration

Session.

16. Hans-Christian Jetter, Werner A. Konig, and Harald Reit-erer. Understanding and designing surface computing with

zoil and squidy. In CHI 2009 Workshop - Multitouch andSurface Computing, 2009.

17. M. Kaltenbrunner, T. Bovermann, R. Bencina, andE. Costanza. Tuio - a protocol for table based tangible userinterfaces. In In Proceedings of the 6th International Work-

shop on Gesture in Human-Computer Interaction and Sim-ulation, 2005.

18. David Kirsh and Paul Maglio. On distinguishing epistemic

from pragmatic action. Cognitive Science, 18(4):513–549,1994.

19. Werner A. Konig, Hans-Joachim Bieg, Toni Schmidt, andHarald Reiterer. Position-independent interaction for large

high-resolution displays. In IHCI’07: Proceedings of IADIS

International Conference on Interfaces and Human Com-puter Interaction 2007, pages 117–125. IADIS Press, 2007.

20. Werner A. Konig, Joachim Bottger, Nikolaus Volzow, and

Harald Reiterer. Laserpointer-interaction between art andscience. In IUI ’08: Proceedings of the 13th international

conference on Intelligent user interfaces, pages 423–424, New

York, NY, USA, 2008. ACM.21. Werner A. Konig, Jens Gerken, Stefan Dierdorf, and Harald

Reiterer. Adaptive pointing: Design and evaluation of a pre-

cision enhancing technique for absolute pointing devices. InInteract 2009: Proceedings of the the twelfth IFIP conference

on Human-Computer Interaction, pages 658–671. Springer,Aug 2009.

22. Jean-Yves Lionel Lawson, Ahmad-Amr Al-Akkad, Jean Van-

derdonckt, and Benoıt Macq. An open source workbench forprototyping multimodal interactions based on off-the-shelf

heterogeneous components. In EICS’09: Proceedings of the

first ACM SIGCHI Symposium on Engineering InteractiveComputing, New York, NY, USA, 2009. ACM.

23. I. Scott MacKenzie and Colin Ware. Lag as a determinant of

human performance in interactive systems. In CHI ’93: Pro-ceedings of the INTERACT ’93 and CHI ’93 conference on

Human factors in computing systems, pages 488–493, New

York, NY, USA, 1993. ACM.24. Brad Myers, Scott E. Hudson, and Randy Pausch. Past,

present, and future of user interface software tools. ACMTrans. Comput.-Hum. Interact., 7(1):3–28, 2000.

25. K. Perlin and D. Fox. Pad: an alternative approach to the

computer interface. In SIGGRAPH ’93: Proceedings of the20th annual conference on Computer graphics and interac-

tive techniques, pages 57–64, New York, NY, USA, 1993.

ACM.26. Thomas Schlomer, Benjamin Poppinga, Niels Henze, and Su-

sanne Boll. Gesture recognition with a wii controller. In TEI

’08: Proceedings of the 2nd international conference on Tan-gible and embedded interaction, pages 11–14, New York, NY,USA, 2008. ACM.

27. M. Serrano, L. Nigay, J.-Y. L. Lawson, A. Ramsay,R. Murray-Smith, and S. Denef. The openinterface frame-

work: a tool for multimodal interaction. In CHI ’08: CHI ’08

extended abstracts on Human factors in computing systems,pages 3501–3506, New York, NY, USA, 2008. ACM.

28. Beat Signer and Moira C. Norrie. Paperpoint: a paper-basedpresentation and interactive paper prototyping tool. In TEI

’07: Proceedings of the 1st international conference on Tan-gible and embedded interaction, pages 57–64, New York, NY,USA, 2007. ACM.

29. Russell M. Taylor, II, Thomas C. Hudson, Adam Seeger,

Hans Weber, Jeffrey Juliano, and Aron T. Helser. Vrpn: adevice-independent, network-transparent vr peripheral sys-

tem. In VRST ’01: Proceedings of the ACM symposium onVirtual reality software and technology, pages 55–61, NewYork, NY, USA, 2001. ACM.

30. V. L. Wallace. The semantics of graphic input devices. SIG-

GRAPH Comput. Graph., 10(1):61–65, 1976.


Interactive Design of Multimodal User Interfaces -...

Documents

Transcript of Interactive Design of Multimodal User Interfaces -...