Dynamic Management of Input/Output Devices for Wearable ...589182/FULLTEXT01.pdfDe Morais W.O....

MASTER2 STIC Spécialité Recherche : Systèmes Embarqués

Dynamic Management of Input/Output Devices for

Wearable Computers

présenté par

Wagner OURIQUE DE MORAIS

Année 2004-2005

Responsable : Jean-Yves TIGLI

EPU de Nice – Sophia Antipolis MASTER STIC, Spécialité SE

930, Route des Colles, BP 145, 06903 Sophia Antipolis Cedex - FRANCE

De Morais W.O. Dynamic Management of Input/Output

Devices for Wearable Computers

of 47

Acknowledgements

At first, God bless us all. I would like to thank my advisor, Jean-Yves Tigli for his guidance, support and encouragement during the preparation of this dissertation and articles.

Words cannot express my appreciation and deepest gratitude to my

father, my mother and my brother for their love and emotional support. Thanks to all my family.

My special thanks to Diane Lingrand for her support and for the

articles. I want to express my gratitude to my friends from Brazil and to my new

friends here in France to the moral support. I would like also to thank Celio Trois for his friendship. Finally, thank for

all people involved during the academic year.

"If the future's looking dark We're the ones who have to shine

If there's no one in control We're the ones who draw the line

Though we live in trying times We're the ones who have to try

Though we know that time has wings We're the ones who have to fly..."

Everyday Glory - Rush



of 47

Contents

Figures List .................................................................................. 4

Tables List ................................................................................... 5

Graphics List ............................................................................... 6

1 Introduction ........................................................................... 7

1.1 Motivations ................................................................................. 8

1.2 Objective .................................................................................... 8

2 Wearable Computers ............................................................ 9

2.1 Definitions .................................................................................. 9

2.2 Challenges of Wearable Computing .......................................... 9

2.2.1 Power ......................................................................................... 9

2.2.2 Heat dissipation ........................................................................ 10

2.2.3 Network .................................................................................... 10

2.2.4 Privacy and Security ................................................................. 11

2.2.5 Wearable devices and interfaces ............................................. 11

2.3 Wearable Input and Output devices ......................................... 12

2.3.1 Wearable Input Devices ........................................................... 12

2.3.2 Wearable Output Devices ......................................................... 18

2.4 Chapter Remarks ..................................................................... 19

3 A Classification Model for Input Devices .......................... 21

3.1 Taxonomies of Input Devices according to the task ................ 21

3.2 Taxonomies of Input Devices according to the properties of the devices ..................................................................................... 23

3.3 Chapter Remarks ..................................................................... 26

4 Models of Interaction with Multimodal and Interactive System ................................................................................. 28

4.1 Perception Process .................................................................. 30

4.2 Action/Control Process ............................................................ 30

4.3 Chapter Remarks ..................................................................... 31

5 Equivalence of Devices According to the Human Factors ... .............................................................................................. 32

5.1 Equivalence at Physical Interaction Level................................ 32

5.2 Equivalence at the User‟s Cognitive Level............................... 33

5.3 Chapter Remarks ..................................................................... 34

6 Conclusion and Future Work ............................................. 37

References ................................................................................. 39



of 47

Figures List Figure 1 - Half Keyboard [71] ..................................................................................... 13

Figure 2 - WristPC Keyboard from L3 Systems [69] .................................................. 13

Figure 3 - FrogPad keyboard [67] ................................................................................ 14

Figure 4 - HandyKey Twiddler 2 [68] ......................................................................... 14

Figure 5 - Unistrokes, Graffiti and Jot alphabet examples .......................................... 15

Figure 6 - A soft keyboard in a touch screen ............................................................... 17

Figure 7 - Finger Trackball .......................................................................................... 17

Figure 8 - Touchglove used in [65].............................................................................. 18

Figure 9 - Starner in [19] wearing the Microoptical [72] head-up display .................. 19

Figure 10 - Adapted taxonomy of Foley et al. [26] ..................................................... 22

Figure 11 - Buxton's taxonomy [16]: property sensed, number of dimensions for each

property and the sensed type ................................................................................ 23

Figure 12 - A mouse defined using the composition operators of Mackinlay et al. [38]

............................................................................................................................. 25

Figure 13 - Reclassification of devices according to Card et al. [18] .......................... 25

Figure 14 - Classification of input devices .................................................................. 26

Figure 15 - The mouse and the Etch-A-Sketch toy characteristics .............................. 27

Figure 16 - Adapted model of Schomaker [53] ........................................................... 28

Figure 17 - Model proposed by Beaudouin-Lafon [8] ................................................. 29

Figure 18 - Model of Human Devices Interactions ..................................................... 32

Figure 19 - The mouse and the Etch-A-Sketch toy Operation ..................................... 33

Figure 20 - Using the mouse in different ways ............................................................ 34

Figure 21 - Schematic representation of ICS from [13]. ............................................. 35



of 47

Tables List

Table 1 - Summary of the taxonomies presented......................................................... 26

Table 2 - Different senses and their corresponding modalities from Schomaker et al.

[53] ....................................................................................................................... 30

Table 3 - The functions of the each subsystems[66].................................................... 36



of 47

Graphics List

Graphic 1 - Number of Input devices according to the evolution of computers .......... 20



of 47

1 Introduction

After the creation of the transistor and the integrated circuit, the microelectronics‟ evolution played the main role in the miniaturization in electronics. The miniaturization of the processors and memories allowed the production of smaller computers. These reduction of dimensions combined with the increased capacity of processors have their effect on the evolution of computers.

Until 1975, many people used a single, huge and centralized

mainframe computer. This period of time is known as the Era of Mainframe computer. From 1975 to 2000, the concept was changed to one computer, desktops and laptops for example, to each individual user. This is the era of Personal Computers.

Nowadays, with the advances in electronics and communication

industry it is possible to imagine numerous computing devices of varying sizes to be embedded in the environment [64]. In this approach, the computing requirements and information will be available to user anywhere, anytime as it is a part of the environment itself. This is expected to be the Era of Ubiquitous, Wearable and Mobile computers.

Most Wearable Computers attach a central processing unit and a

battery pack to a belt or to a back pack. More portable keyboards can be worn on the wrist and smaller ones, known as chording keyboards, can be held in the palm of the hand. The visual output is done by devices that are worn as eyeglasses. A tiny screen projects an image onto a silicon lens mirror, which reflects into the user‟s eye.

One of the most important challenges in wearable computing is to

provide input and output devices that balance portability, usability and unobtrusiveness requirements.

This research is concentrated on wearable computers and the main

focus will be how managing dynamically wearable input and output devices without having to modify the main software application [74] [75].

The traditional mouse and keyboard are the most common input

devices and they become a standard method for input for the great part of the software applications. However, they can't be practical or appropriate for the wearable computing.

There are a lot of alternative input devices, but they are only used

when the task requires a device with more precision or when the user has some disabilities or difficulties to use the mouse or the keyboard.



of 47

1.1 Motivations

Wearable devices allow users to wear their devices in a personalized way. Then, they can use theses devices to interact with desktop applications. For example, if the user goes study abroad, he can find some difficulties or decreasing in performance while using a different keyboard layout. If he is carrying a chord keyboard, he can interact with applications in a personalized way.

We agree that in the future, we will assist to a generalization of

wearable computers use. It implies that most common input devices, the keyboard and mouse, have to be replaced by wearable input devices.

We also agree that developing new devices, the designer needs to test

and evaluate these new devices. However, it is necessary that the user's applications work with a given device without having to be modified.

1.2 Objective

The objective of the research reported here is to propose a model of

equivalence between input devices. Users can carry devices like chord keyboards, finger trackball and other wearable devices. However, some times theses devices may not work or be appropriate for a given applications. We want through our model provide an efficient way to establish an equivalence between new input devices and the common input devices, the keyboard and the mouse.

This model will serve as a foundation for mapping events from a given

device to the standard keyboard's and mouse's events. The equivalence model can help the development of interactive and multimodal applications. With this model, an advanced user can build an input device and test, evaluate and compare it using the same main software application.

For this, in the Chapter 2 we will introduce the definitions, challenges

and input and output devices of wearable computers. After, in the Chapter 3 we will present a survey of different taxonomies for input devices that guided the development of our first model.

In the Chapter 4, we will present some model of interaction with

multimodal and interactive systems. In the Chapter 5, we show the importance of considering the human‟s factors in the development of a model of equivalence between devices. This chapter is followed by the conclusion and future work that can be made in our work.



of 47

2 Wearable Computers

The miniaturization of computers and sensory devices allows humans to wear these devices and the major topic of wearable computing research is to smartly assist humans in daily life [54].

Wearable computing can improve productivity because it provides

constant access to computing and communication resources to the user allowing to do more complex tasks in less time and with less effort [9] [39] [40]. However, there are a lot of challenges that must be overcome.

This chapter introduces the Wearable Computer: definitions, challenges and wearable input and output devices.

2.1 Definitions

According to Steve Mann a wearable computer is a computer that is

subsumed into the personal space of the user, controlled by the user, and has both operational and interactional constancy, i.e. is always on and always accessible [40].

Starner [59] from the Georgia Institute of Technology defines wearable

computing as: “Wearable Computing pursues an interface ideal of a continuously worn, intelligent assistant that augments memory, intellect, creativity, communication and physical senses and abilities”. Bradley Rhodes in [49] defines the wearable computer by its

characteristics that are: portability while operational; hand-free use; sense the user's context; be able to convey information to its user even when not actively being used and finally be always on.

2.2 Challenges of Wearable Computing

In order to provide wearable computers, some challenges must be overcome [19] [59] [60]. Challenges such power, heat dissipation, networking, and wearable devices and user‟s interfaces are discussed in [59] [60].

2.2.1 Power

To satisfy the always on requirement, cited in the section 2.1, the

challenge is provide to the wearable computer with a continuous source of power. Power management is a one of the most crucial considerations when designing a wearable system.



of 47

Adding more features to the wearable computer, like a faster

processor, a larger disk, a wireless network connection and so on, requires more power, which increases the battery's size and weight [19].

In [59], Starner states that before designing the functionality or

packaging of a device, the designer must first decide factors like cost, size and weight for the battery.

The power consumption of a system depends on the system capacity,

the operating system employed, board and modules used, and the physical environment that the unit is exposed to [20]. For example, a wearable system uses multimodal interfaces, combining speech and gesture recognition for a given task, the processing power and the battery life are critical issues [43].

Solutions for these problems could be small batteries with long lifetime,

rechargeable batteries that can be recharged using the body motion or trough radio transmissions. However, these solutions have some problems like using radioactive materials or having to wear an antenna [59].

2.2.2 Heat dissipation

Designed to embedding into or like user's clothing, the wearable

computer is in contact or close to the user's body during the most part of the day. This proximity limit the amount of heat that a device may generate and should note exceeds 40° C [60].

Starner in [60] and Starner et al. [58] present some examples of how

heat dissipation can be managed. Using the movement of the user, that create some airflows, one could improve the heat dissipation of an arm-mounted computer. Another, is using materials that absorb the heat while they maintain the same temperature as they transition form solid-to-liquid or liquid-to-gaseous phases. Batching and delaying jobs until the environment became cooler, is a software alternative to the problems of heat dissipation.

2.2.3 Network

"For wearable computers, networking involves communication off body to the fixed network, on body among devices, and near body with objects near the user" [59].

Off-body communications means the communication with objects in the

environment. The most common technology used is the same of the cellular phones use. The problem here is that some times the mobile device will not be in range of a network cell.



of 47

On-body communication means the network between user's devices. The technologies that are often used are Bluetooth and IEEE 802.15. However, energy use becomes critical because each device must have its own battery. According to Starner [59], a Body LAN system that use very little power to communicate, can be a way to approach this problem. Another issue pointed by him, is the privacy when different on-body devices communicate to each other.

In the communication with near-body objects, the issue is that the

standards used by electronics manufactures are based on radio and infrared transceivers. These standards assume that the device is powered by significant energy supply. An alternative is to use radio frequency identification (RFID) tags and tag readers [59].

2.2.4 Privacy and Security

There is an important remark that must to be made that is differs

privacy from security. In [59], there is an example using an active-badge system that a company deploys as a security measure. The system is used to control the access to the rooms. In the point of view of security, it is efficient. However, it raises several privacy concerns, because the employees can be monitored, for example, one can see how long time someone spent in the coffee break or in the restroom.

Wearable computers allow users to have access to information that

they normally wouldn't have [9] [40]. The user's information is stored in the wearable computer, instead of in central databases [60]. Users can record conversations, keep personal notes, schedule, diary and medical records on their wearable. When having that type of concentration of information in one place, that means somebody who got hold of that information would know a lot about the user. So, the users have to use a combination of security measures like encryption, physical measures like keeping the machine on them, and guarding it like a purse to protect it.

2.2.5 Wearable devices and interfaces

Advances in electronics have resulted in rapid miniaturization of

computing and mobile communication devices [27] [34]. However, one of the most important challenges is the miniaturization of the user interface of these devices.

Schmidt et al. [51] propose a system using RFIDs and a wearable tag

reader that enable implicit human computer interaction using the user activity in the real world as input to computers. However, in an explicit interaction, providing input to wearable computers is difficult, especially when users are not stationary [51]. The traditional keyboard is not practical while moving, so the user must choose an alternative input method.



of 47

For text input, he can use chord keyboards, half QWERTY forearm mounted keyboards, virtual keyboards or speech recognition. To use a chord keyboard, the user has to be trained over a rather long time. There is also the issue of the cognitive load of inputting text while doing something else [52]. Speech recognition is another option, but there are some problems in the accuracy of the recognition system.

For pointing, different devices are proposed for wearable computers.

The mouse has been replaced by trackballs, touch pads and other more mobile pointers devices. All of these devices are small, but still large enough to impose limits on the size of the mobile device. If the user is sitting or standing still, the pointing accuracy of these pointing devices is acceptable, but if the users is walking or doing something else these devices is not useful to operate standard graphical user interfaces based applications.

Applications that have been designed for desktop use, do not meet the

needs for wearable use. The desktop graphical user interface is based on the WIMP (Window Icon Menu Pointer) metaphor and assumes that the primary task is the interaction with the computer [9]. In comparison with it, the interaction with the wearable computer is not the primary task. The wearable computer is used as a tool.

2.3 Wearable Input and Output devices

In wearable computing it is difficult to provide inputs to the system [52]. Nowadays, the mobile computing solutions are still complex and obtrusive. Depending on the context the user can change a device for another one that best fits the task requirements. After finish the task, he can change again, if the current device is not appropriate for the new activity.

2.3.1 Wearable Input Devices

For decades the standard devices to transfer information from a user to

the computer have been a mouse and a keyboard. However, the keyboard is not portable and require, as a mouse, supporting surface. Nowadays, the mobile computing solutions are still complex and obtrusive.

Input devices for a wearable computer must offer hands free or one

hand operation. They must also balance the input speed, accuracy, small physical form factor, portability, be cheap and be easy to use, requiring a little learning time. In addition, these devices should not require extensive computing resources that will not be available in low-weight, low-energy wearable computers.

To provide a less obstructive way to input text to wearable computers,

the user has a range of alternative choices that include Half-Qwerty, chord and soft keyboards, speech and handwrite recognition and others. Some



of 47

more sophisticate method can be made using video cameras or instrumented gloves.

The Half-Qwerty keyboard [41] is a QWERTY keyboard that is split in

half. To enter characters the user types the appropriate key, but if the space bar is held while a key is typed, the corresponding character from the other half of the keyboard is entered. Hitting the space bar alone types a space. This device is operated using one hand and either hand can be used. The half-qwerty keyboard can be mounted on the forearm as showed in the Figure 1.

Figure 1 - Half Keyboard [71]

In the same class of reduced keyboards, we can find the WristPC [69].

The WristPC is a reduced Keyboard that was designed for portable and wearable computer applications. Its layout is optimized to provide alphanumeric entry. It can operate in raining and others environments. A curved back provides a secure and comfortable placement on the wrist as presented in the Figure 2.

Figure 2 - WristPC Keyboard from L3 Systems [69]



of 47

The FrogPad [67] is another reduced one-handed keyboard presented in the Figure 3. It provides all keyboard functionalities, allowing users entering alphanumeric, punctuation, symbols and even function keys.

Like using a chord keyboards, the user must learn and practice to

familiarize with the FrogPad layout. However, unlike chording keyboards, the Frogpad can produce any character without ever having to press more then two keys at a time.

Figure 3 - FrogPad keyboard [67]

Other possibility for text input is using a Chord keyboard. It was

designed to input text data using a reduced set of keys. To enter a character, the user must press several keys simultaneously. A chord keyboard can be held in palm of the hand, don't need a surface to mount it on and requires low power.

As a chord keyboard can be operated with one hand and allowing the

user enters text while moving, it is much utilized in wearable systems [37] [62]. Moreover, blind and some disabled users can use a chord keyboard easily. However, to use this kind of alternative keyboard, the user must spend some time learning the key map also training. The disadvantage of the chording keyboard is that it cannot be without some training before use.

The Figure 4 shows the HandyKey Twiddler 2 [68], a one-handed

chord keyboard and mouse in the same device.

Figure 4 - HandyKey Twiddler 2 [68]



of 47

Another method for text input is Handwriting Recognition. As users know how to write, a handwriting recognition system can be easy to use. However, to offer good results, for example, writing a cursive paragraph with only one or two recognition mistakes, a handwriting recognition system demands a high computation [57]. Even more, to facilitate computer recognition, the user must learn a special alphabet like Unistrokes, Graffiti for Palm OS [73] and Jot for Microsoft Windows CE [70].

Unistrokes is a single-stroke alphabet developed by Goldberg et al.

[30] that is both easier for software to recognize, and faster for users to write. The Graffiti alphabet, unlike as Unistrokes, has strokes for punctuation,

numbers, symbolic characters, and capital and lowercase switches. Other advantage over Unistrokes is its similarity to normal hand-printed characters.

Nowadays, the Microsoft Windows CE devices uses the Jot alphabet

that recognizes many of the Graffiti strokes, strokes for numbers, symbolic characters, capital and lowercase and allows users to indicate writing preferences for some characters. Some strokes of each alphabet are showed in Figure 5.

Figure 5 - Unistrokes, Graffiti and Jot alphabet examples

For a wearable system, speech recognition is the ideal input method. It provides a user-friendly, flexible, unobtrusive and efficient interface method [36]. If the user‟s hands and eyes are busy, speech may be a good choice for input information. A microphone is small, very low weight and keeps the hands free.

A voice recognition system can be used almost any environment, offer

a faster method than other input devices. It can be used for text and position input, allowing users to input commands or dictate a text [61].

Some voice recognition systems require separate each word with a

short pause. This style of dictation is called discrete speech. Other systems, allow continuous speech voice recognition. With this, a user can dictates text in a flowing conversational tone.



of 47

Over the past few years, a lot of improvements were made in the accuracy of voice recognition [42]. However, a speech recognition system has some problems. One issue is capturing the sound. Some speech recognition systems maintain a huge amount of data. So, there is a recognition problem because is difficult to differentiate some words. Noise, continuing speaking, slang and vocal variations contributes to this problem [61].

Speech recognition allow input data while moving, driving and others

situations [50]. However, particularly in public, it may not always be adequate because it may present a problem of privacy [59] [60].

Stäger et al. [56] present a reduced sound recognition system focused

on an ultra low power hardware implementation. They also present a VHDL model of the hardware showing that their method can be implemented with minimal resources.

Other more sophisticated systems, allow inputting text using gesture

recognition. Most gesture-recognition systems receive input through an external camera or through a sensing glove on the user‟s hand.

Gesture-recognition based on visual input, e.g., using a camera, allows

inputting commands by recognition of movements and facial expressions. However, capturing images of the body requires a certain distance for positioning the image capture which decreases wearability [28].

Placing accelerometers in the finger, one can build a data glove that

gives the ability to recognize hand gestures [48]. They typically report the position hands in three dimensional spaces, and more importantly, the position of the fingers relative to the hand or palm.

Alternative pointing devices, such touchscreen, finger trackballs, touch

pads and other devices are proposed to replace the mouse in the wearable computing.

Touch screens are direct control pointing devices that allow the user,

by touching an appropriate part of the screen, input information into the computer. A touch screen is a bi-directional device, it means that it receives input and it outputs information.

Using a touchscreen, a virtual QWERTY keyboard layout can be

implemented in software, showing the keys on the touchscreen. Kölsch et al. [36] define a virtual keyboard as a touch-typing device that does not have a physical manifestation of the sensing areas. That is, the sensing area which acts as a button is not such as a button but instead is programmed to act as one.

A Virtual keyboard that employs discrete sensing areas for each

symbol inherently allows the realization of a soft keyboard and is presented in the Figure 6.



of 47

Figure 6 - A soft keyboard in a touch screen

The advantage of a soft keyboard over a standard keyboard is that the

size, shape and layout can be changed to fit the user‟s personal, for example, for a small size touchscreen a stylus can be used to type the keys. Stylus tapping on a soft keyboard represents a fast and easy alternative for text entry [55]. However, there are feedback and ergonomic issues. In a touchscreen, there is a lack of tactile feedback. The user must look to the screen constantly to know if he typed the correct key. Moreover, if one wants to enter a great amount of text in a small size touchscreen with a stylus, this task can be very tiresome.

Other alternative device for wearable is the Finger Trackball. It is a two

dimensional indirect pointing device designed to mobile and wearable applications. The user holds it in the palm of the hand using the index finger, moving the ball and using the buttons with the thumb finger as presented in the Figure 7.

Figure 7 - Finger Trackball

A touchpad also can be used as a pointing device in wearable systems

[10]. Devices such TouchGlove [10] consists of a touch-sensitive surface device which is mounted on the palm portion of the half-glove. TouchGlove device from [65] is showed in the Figure 8.



of 47

Figure 8 - Touchglove used in [65]

2.3.2 Wearable Output Devices

As input devices, output devices must also be portable and unobtrusive

[28]. Screen size is limited as the devices must be small enough to be easily carried [15]. In order to provide output to the user, some wearable devices such head-mounted displays, earphones and tactile displays can be used.

Head-mounted displays provide a visual output and can be divided in

optical see through and video see through head-mounted displays [5] [52]. The first type, the output images are projected onto the lens in front of

the user's yes. This projection is usually done only on one of the lens, leaving the other lens free for clear vision [5]. This type is more used in wearable computing.

The second type requires full user's attention. Video see through head-

mounted displays use opaque screens in front of the user's eyes and are combined with head-mounted video cameras [5]. The real-world view captured by the cameras is combined with the computer generated objects and is projected on the opaque screens. The real-world view quality depends of the quality of the cameras. This kind of head-mounted displays is often used in virtual reality applications.

Head-mounted displays have some issues like limited resolution,

ergonomics, obtrusiveness and others. Some advances in embedding the display into glasses by Microoptical Inc. [72] make head mounted displays socially acceptable as showed in the Figure 9.



of 47

Figure 9 - Starner in [19] wearing the Microoptical [72] head-up display

Output information can be also provided by audio. Audio output can be provided by speech synthesis systems that take a string of characters and creates spoken output. Otherwise, the output can be done by beeps or a particular sound to alert the user.

When the user's eyes are busy or the application doesn't output much

data or to signal some events an audio output can be a good choice [50]. Earphones are not obstructive and are a suitable device for a fulltime-wear environment [28].

Such as audio, tactile displays are presented as alternatives to visual

displays. Moreover, it can be used when the user are eyes and ear busy. Tactile displays are unobtrusive and do not require a user‟s visual attention. An example of this is the vibrotactile displays that are used in mobile telephones.

Brewster [15] states that tactile displays can be combined with audio

and visual output to create fully multi-modal displays.

2.4 Chapter Remarks

This chapter, we presented a new family of computers, the wearable

computer. By the definitions, we can see that a wearable computer differs from a mobile device like a cell phone, PDA and others portable devices because the user doesn‟t need to stop his task and take the device to use it.

As pointed by [12] [19] [60], one of the most important challenges is to

provide various devices for input and output that balance portability, usability and unobtrusiveness requirements.

We also presented some examples of wearable input and output

devices. Crabtree et al. [19] state that that a good wearable computing design depends greatly on the particular applications intended for the device.

Moreover, depending on the user context, some devices could not be appropriate [46] [56]. For example, speech recognition is the better choice



of 47

while driven, but when users want privacy in public, they must use an alternative keyboard.

According to Moeslund et al. [43] if all the devices could communicate

and share a common set of input and output devices the quality of interaction could be greatly improved. The Graphic 1 presents the number of input devices used by the user according to the evolution of computers. As we can see, wearable computing opens new directions in the research of input/output devices and interfaces.

Graphic 1 - Number of Input devices according to the evolution of computers

Then, we propose a classification model for input devices to provide an

efficient way to select and use different devices without having to change the main software application [74] [75].



of 47

3 A Classification Model for Input Devices

In a traditional wearable computer configuration, the user can carry input devices like a chord keyboard, a finger trackball, a touch tablet, a joystick and so on. The user can choose, among the devices, the best one that is more appropriate to do some task. However, the software application, for using these or other input devices, must typically be modified for each device.

For Jacob et al. [32], a useful approach for this situation is making an

input device taxonomy and design user interfaces that respect this taxonomy. This approach turn it possible to determine when a given device should be used or how it can replaced for another one.

Since 1970, a lot of taxonomies for input devices were proposed, trying

to provide a mechanism for understanding and discussing the similarities and differences among devices [14].

3.1 Taxonomies of Input Devices according to the task

Foley et al. [25] propose to separate the input task from the input

device. They created four virtual devices, the button, pick, locator, and valuator those are independents of the input device.

This approach would allow software developers to support many input

devices and make the application more portable because they would code for generic devices rather than specific devices.

In addition to establishing the notion of virtual devices, they identified

the importance of decomposing interactive tasks into lexical, syntactic and semantic levels and pointed out the pitfalls of incorrectly processing input at one level that was intended for a different level [38].

Other approach, by Enderle et al. [22], refined the taxonomy above

adding two others virtual devices, the stroke and string. These virtual devices have the following characteristics:

choice: designate system defined objects;

pick: designate user defined objects within the software system;

locator: determine position and/or orientation;

valuator: determine a single value within some number space;

stroke: sequence of (x, y) coordinates;

string: sequence of characters. These taxonomies, allow flexibility and rapid prototyping of graphic

software applications, allowing the use of different devices without modifying the application, only the device driver [17]. However, some devices that are



of 47

equivalents according to the classification can be very different both physically and practically [14].

In another work, Foley et al. [26] decided to map the user's intentions

(select, position, orient, path, quantify and text) to the device that can performs these tasks. They enumerated a relatively comprehensive set of techniques and technologies capable of articulating each of these basic primitives as showed in the Figure 10.

Figure 10 - Adapted taxonomy of Foley et al. [26]

In this approach, they hide the crucial pragmatic aspects of haptic input by treating devices that output the same information as equivalent, for example, selecting with a joystick or trackball equivalent, despite the different subjective qualities that the devices present to the user [33].

Thus, the taxonomies cited above are not sufficient to us in order to

establish a model of equivalence between devices.



of 47

3.2 Taxonomies of Input Devices according to the properties of the devices

In 1983, other taxonomy was proposed by Willian Buxton [16] in terms

of input devices properties, like:

property sensed (position, motion and pressure);

the number of dimensions for each property sensed (for example, a mouse returns two values for the position property);

sensing type which distinguishes between devices that can be touch (direct manipulation, for example, the touch screen) or mechanical (indirect manipulation, for example, the mouse).

Traditional pointing devices typically sense position, motion, or force. A

tablet senses position, a mouse measures motion, an isometric joystick senses force, and to a rotary device the corresponding properties are angle and torque [31].

For each physical property sensed, the device can measures one or

more linear and angular dimensions. For example, a knob or rotary pot measures one angular dimension, a mouse measures two linear dimensions and a six degree of freedom magnetic tracker measures three linear and three angular dimensions.

In the Figure 11, some devices are placed in a matrix. The rows and

columns represented by solid lines identify the properties sensed and the number of dimensions sensed. For example, potentiometers are one dimensional devices and are placed in the left column but a mouse is a two dimensional device and are placed in the center column. Trackballs sense motion and are placed in the center row but isometric joysticks are pressure sensing and are placed in the bottom row.

Figure 11 - Buxton's taxonomy [16]: property sensed, number of dimensions for each property and the sensed type



of 47

The rows and columns represented by dashed lines delimit devices manipulated using different motor skills and devices operated by direct touch or a mechanical intermediary. These characteristics are placed in sub-columns and sub-rows, respectively. For example, potentiometers may be rotary or linear and are placed in the left sub-columns. Screen input may be direct through touch or indirect through a light pen and are placed in the top sub-rows.

Buxton [16] points out that this taxonomy can help to finding

appropriate equivalencies between devices, makes it easy to relate different devices in terms of metaphor and is useful in helping quantify the generality of various physical devices. However, the main weakness of this taxonomy is that it only considers continuous and manually operated input devices [38].

In [16], Buxton advocated that input devices are transducers of

physical properties in one, two, or three dimensions. Extending the Buxton's idea, mainly in the notion of the physical property sensed, Mackinlay et al. [38] made a distinction between:

absolute and relative quantities;

translational and rotational movement;

the axes in which the device operates. The dimensionality of a device is indicated by composing the axes in which it operates;

numbers of values that a device can sense in a particular axis. They made an example using a radio with three knobs. Through this

example, they illustrate some important parts of an input device: the geometry of transducers of physical manipulation; the domain of values that the transducer can produce; device resolution; and connections among devices.

According to Mackinlay et al. [38], an input device can be defined as a

6-tuple < M, In, S, R, Out, W >, where each component is defined as follow:

M is a manipulation operator(physical property sensed);

In is the input domain set;

S is the current state of the device;

R is a resolution function that maps from the input domain set to the output domain set;

Out is the output domain set;

W allows describing how the input device works. They also defined composition operators, the merge composition,

layout composition and connect composition. These operators allow the description of interaction techniques and combination devices. The Figure 12 presents the composition that describes a mouse.



of 47

Figure 12 - A mouse defined using the composition operators of Mackinlay et al. [38]

In [18], Card et al. pointed out the coverage of this taxonomy,

reclassifying the devices listed by Foley et al. [26], Buxton [16] and others. This reclassification is showed in the Figure 13.

Figure 13 - Reclassification of devices according to Card et al. [18]

Using the ideas of each taxonomy showed above, we created a tree-

based model that can help us to separate devices that provide text input from the pointing input. This model is showed in the Figure 14 where we can see different devices and a possible correspondence with the keyboard or the mouse.



of 47

Figure 14 - Classification of input devices

3.3 Chapter Remarks

In this chapter we presented our survey about the existing taxonomies

for input devices. Each of the taxonomies is summarized in the Table 1. Taxonomy Characteristics Advantages Drawbacks

Foley et al. [25]

Input task independent of the input device; Four virtual devices (pick, button, locator and valuator)

Allow discussing an input model that is independent of the hardware

Don't considerate that a given devices is more appropriate than other to a particular task

Enderle et al. [22]

Six virtual devices (pick, choice, locator, valuator, stroke and string)

Device-independent input library

Same problem of the taxonomy of Foley et al. [25]

Foley et al. [26]

For a given task, a list of devices that can be used

Knowing the task, the taxonomy show the appropriate device

Limited set of devices and many devices can be used to do a same given task; Is necessary to map all tasks, all devices and all interactions. Adding new devices, the model must be changed

Buxton [16] Classification according to the properties sensed (position, motion and pressure);number of dimensions and sensed type

Consider the properties of the device; Consider the human motor and sensory system

Don't support discrete devices neither voice input

Mackinlay et al. [38]

Notation to describe the composition of devices Compositional Operators (merge, layout, connect)

Classifies devices both physical and virtual

Don‟t support voice input

Table 1 - Summary of the taxonomies presented



of 47

The taxonomies cited above allow comparing and designing new devices. Moreover, they allowed us to build a classification model for input devices.

Both the taxonomies and our classification of input devices present a

functional equivalence between devices. To illustrate this, we present an example using the mouse and the Etch-A-Sketch toy. The Figure 15 shows the devices.

Figure 15 - The mouse and the Etch-A-Sketch toy characteristics

Both of the devices can be used to control the cursor position on a screen and are composed by two one-dimensional sensors. The difference between these two devices is that the Etch-A-Sketch toy can be controlled by the user using rotary pots in separate dimensions, i.e., he can control the x axis independently of the y axis. For the mouse, in contrast, x and y axis are mechanically coupled.

As can be seen, both of the devices, when manipulated, return a two

dimension position. It means that they are functionally equivalent. Let us try to draw an oblique line: it is really easier using a mouse instead of the Etch-A-Sketch.

Thus, a model of equivalence based only on the functional

equivalence, i.e. devices that output the same information, is not sufficient for our purpose. We then propose to create a model of equivalence between devices that takes into consideration the interaction with the user and the user context.



of 47

4 Models of Interaction with Multimodal and Interactive System

The taxonomies of input devices presented before consider the user‟s task and the devices' properties. However, a model of equivalence must also consider the human capabilities and limitations.

In an interactive system, both the user and the computer interpret each

events occurring during the interaction. There is a continuous dialog between the user and the computer. The user produces information through input devices to the computer. The computer processes this information and returns the output result to the user. This output information will determine the next interaction of the user. This means that the user will evaluate the resulted information before another action.

A lot of works have been done around the current state-of-the-art of

human-computer interaction as can be found in [21] [32] [54] and also the interaction with multimodal [11] [44] [45] [53], ubiquitous computing [4] [23], wearable computers [3] [62] and virtual reality environments [29].

Some relevant models of multimodal human-computer interaction and

interactive systems are showed by Schomaker et al. [53] and Beaudouin-Lafon [7], respectively.

Schomaker et al. [53] distinguish four categories of input and output

channels. The Figure 16 shows an adapted model for the basic processes in human-computer interaction.

Figure 16 - Adapted model of Schomaker [53]



of 47

The Human Output Channels and Computer Input Modalities define the input flow, while the Computer Output Media and Human Input Channels define the feedback flow. The input flow and the feedback flow allow a complete interaction information flow.

Beaudouin-Lafon [7] decomposes an interactive system in two parts and proposes a generic conceptual model for interactive systems. The two parts are the user interface and the functional core. The user interface is composed by hardware devices and software applications that are used to get the user input and to retrieve the output information of the system. The functional core corresponds to the internal functions of the system.

The model proposed by him is presented in the Figure 17 and is

composed by five components:

Objects: correspond to the objects of the domain application (e.g. words and sections in a text editor, etc) that are accessed and modified by operations;

Operations: internal operations that deal with objects and are activated by user‟s commands;

Commands: represent instantaneous user‟s actions (e.g. select an object) or can be composed by elementary actions (e.g. drag-and-drop of a file to a folder);

Feedback: corresponds to the expectation that the user has while doing some action (cursor position while moving the mouse) or the state of the system (sound indicating a warning information);

Responses: represent the commands‟ results that are recognized by the user.

Figure 17 - Model proposed by Beaudouin-Lafon [8]

Both, Schomaker et al. [53] and Beaudouin-Lafon [7] identify two processes involved at the human side of the interaction that are the perception and control. The perception process refers to the human senses to percept the computer„s output information. Control process represents the user‟s interaction, by physical movements or voice, with the computer‟s input devices.



of 47

4.1 Perception Process

Perception involves the use of our senses to detect information.

Human senses like sight, hearing, touch, smell, taste and balance allow the perception of the environment. The Table 2 presents the association among the human‟s sensory system and the respective modality.

Sensory perception Sense organ Modality

Sense of sight Eyes Visual Sense of hearing Ears Auditive Sense of touch Skin Tactile Sense of smell Nose Olfactory Sense of taste Tongue Gustatory Sense of balance Organ of equilibrium Vestibular

Table 2 - Different senses and their corresponding modalities from Schomaker et al. [53]

In a human machine interaction, the main perceptional channels are

the visual, auditive and tactile modalities. The information received through the visual channel is much larger than those other channels.

Currently, the visual channel provides most of the sensory feedback in

the task space of a graphical user interface [2]. However, it is easy to imagine situations in which it is not appropriate receive information on the screen. If we receive an email while driving, the best and secure way to present the information would be if the computer reads the message.

The motion or vibration of objects produces rapid changes in the air

pressure that are detected by the middle ear. This is the process of sound detection. The auditive perception is indispensable in some contexts where the user is with his hands and eyes busy [50].

Through haptic interfaces, such as force feedback and tactile devices,

users can receive information by the sense of touch in a non-visual manner. The tactile perception is also associated to the proprioception and kinesthesia [7]. Proprioception is the ability to sense the position, orientation and movement of the body and its parts.

Kinesthesia is the ability to feel the movements of the limbs and body.

It allows for example the eye-hand coordination. For example, we don‟t need to pay attention in our hand while moving the mouse.

4.2 Action/Control Process

Voice and physical movements allow interacting with the environment

and are the main modalities for the action/control process. Voice is the



of 47

primary means of communication between humans. However, there are some issues in full control of the computers using voice [42] [61].

In the human body, bones are connected by joints. Some joints allow

abduct and adduct, flex and extend, and rotate. Ball and socket joints such in the shoulder allow three degrees of freedom. Human hands are the most dexterous body parts and are heavily used in both manipulation and communication [63]. Most input devices rely on hand movements. However, impaired people can use eyes, foots and mouth to manipulate special devices.

4.3 Chapter Remarks

In this chapter we presented some models of the human-machine interaction in multimodal and interactive systems. In the models of Schomaker et al. [53] (Figure 16) and Beaudouin-Lafon [7] (Figure 17) is it possible to see that there are three loops that involve the interaction with the computer and its devices.

The first loop considers the user and the computer. It represents that

the user‟s input information will be evaluated and computed according with a well defined logic and will be returned to the user. Then, the user evaluates this response and determines the next interaction.

The second loop represents the logical feedback of the user’s action on

the devices. While moving the mouse, the user receives visual feedback that is the movement of the cursor across screen. Text appearing while the user is typing in the keyboard or sounds indicating a warning is other examples.

The third loop involves only the user and the input device that he is

using. For example, the haptic feedback gave by the mouse button when clicked. Other example is that a typist doesn‟t need to look to keyboard while typing. Since he knows the keys positions, he can detect when made a key mistake.

Currently, the feedback information is mainly provided to the visual

perception. Tactile feedback incorporates the sense of touch in the interaction and the auditory feedback is often used as an extension of visual feedback. Thus, we saw that there is a physical interaction with the computer and their devices. Also, we identified the cognitive aspect involved in the interaction with the input device. Then, we want to improve our model of equivalence considering the human‟s factors that involves their physical and cognitive capabilities [74] [75].



of 47

5 Equivalence of Devices According to the Human Factors

Humans have different physical and cognitive capabilities, as well as

different skills and experiences. With the models and concepts in Chapter 4, we build a model that summarizes the interaction process between device and user as represented in Figure 18. In this model we distinguish physical and cognitive capabilities required in the interaction process.

Figure 18 - Model of Human Devices Interactions

Through this model, we want to present that the physical actions and

the cognitive process during the interaction can determine or not the equivalence of devices.

5.1 Equivalence at Physical Interaction Level

Two different devices that are used for the same purpose, pointing for

example, can be equivalents at the physical level if the user performs the same action for both devices.

Then, we go back to example of the mouse and the Etch-A-Sketch toy

to present that different pointing devices are not equivalent in the physical level of the interaction. The Figure 19 presents our redefined example.



of 47

Figure 19 - The mouse and the Etch-A-Sketch toy Operation

The user manipulates the mouse holding the palm of the hand on the

mouse and moving it in two dimensions, using the combination of the elbow and shoulder's degrees of freedom.

To control the Etch-A-Sketch, the user uses two hands, holding each

rotary pot using the thumb and index fingers. To rotate a rotary pot, he moves each finger in one dimensional space. To draw diagonal lines, the user must coordinate his movements. It means, doing synchronized horary or anti-horary rotations.

Drawing ellipses and diagonal lines with the Etch-A-Sketch is hard and

it is necessary a long time of training. However, is easier to draw perfect line using the Etch-A-Sketch than using the mouse.

Then, even allowing the user moving the cursor across the screen, the

Etch-A-Sketch is not equivalent to the mouse in the physical level of interaction.

5.2 Equivalence at the User’s Cognitive Level

Cognition is the process by which people gain acknowledgment. The

cognitive process involves understanding, remembering, reasoning and creating new ideas for something [7] [53]. At the cognitive level we find the mental process that is responsible to determine our actions according to our perception.

According to Buxton et al. [17], the use of computing systems makes

heavy demands on our cognitive system. Performing many tasks imposes a high cognitive load.



of 47

Then, we want to show that even using the same device in a different way represent a non equivalence at the cognitive level. Considering that the user inverted the position of the mouse as presented in the Figure 20.

Figure 20 - Using the mouse in different ways

Using the mouse in the inverted position can turn a simple task, such

selecting an icon in the screen, a really challenge. Since the cursor movement is opposite to the physical movement on the device, the user must keep a mental process in to manipulate de device in the right way.

In order to obtain the same or almost the same performance of using

the mouse in the right way, the user must practice and repeat the task several times, in order to improve the perception and the cognitive system.

5.3 Chapter Remarks

As we can see, another point that has to be considered in build a

model of equivalence between input devices is the human’s physical resources and the metal process involved in the interaction. Both of these levels allow differentiate devices that are used for the same purpose. The main aspect of the difference can be noted in the usability and performance.

Kabbash et al. [35] compared the use‟s performance using mouse,

trackball, and tablet-with-stylus in the preferred and non-preferred hands. They concluded that the non-preferred hand can be used in tasks that do not demand precision, such as scrolling. According to Kölsch et al. [36] the speed of classic touch-typing with ten fingers is limited by cognitive skills rather than by the motor skills required to independently and rapidly move the ten fingers. For pen-based interfaces the speed is limited by motor skills rather than by cognitive skills.



of 47

To evaluate the user performance while using some device, we can use two techniques. The first one is the Fitts’ Law [24]. The Fitts‟s law is used calculate the movement time for pointing. The main interest of Fitts‟ law is that it allows the translation of the performance scores from different devices into indexes of performance, which are independent from the experimental conditions used in the different tests, allowing a direct comparison of the devices [47].

The second one is the Steering Law [1] that describes constrained

movement performance. It is used for the evaluation of trajectory movement tasks based on constrained motion for different path shapes [47].

In order to verify the equivalence at the cognitive level, we want to

adopt the Interacting Cognitive Subsystem (ICS) [6] model. In this model, the human information processing system is subdivided into a set of specialized subsystems, where each one is responsible for represent and deal with specific types of information.

Figure 21 - Schematic representation of ICS from [13].

The Figure 21 presents the schematic representation of ICS and the Table 3 summarizes each subsystem.

We believe that we can build a model of equivalence between devices

analyzing each subsystem during the interaction with the device.



of 47

Peripheral Subsystem

Sensory

Acoustic (AC)

Sound frequency (pitch), timbre, intensity etc. Subjectively, what we „hear in the world‟.

Visual (VIS)

Light wavelength, brightness over visual space etc. Subjectively, what we „see in the world‟ as patterns of shapes and colors.

Body State (BS)

Type of stimulation (e.g., cutaneous pressure, temperature, olfactory, muscle tension), its location, intensity etc. Subjectively, bodily sensations of pressure, pain, positions of parts of the body, as well as tastes and smells etc.

Effector

Articulatory

(ART)

Force, target positions and timing of articulatory musculatures (e.g., place of articulation). Subjectively, our experience of subvocal speech output.

Limb (LIM)

Force, target positions and timing of skeletal musculatures. Subjectively, „mental‟ physical movement.

Central Subsystem

Structural

Morphonolexical (MPL)

An abstract structural description of entities and relationships in sound space. Dominated by speech forms, where it conveys a surface structure description of the identity of words, their status, order and the form of boundaries between them. Subjectively, what we „hear in the head‟, our mental „voice‟.

Object (OBJ)

An abstract structural description of entities and relationships in visual space, conveying the attributes and identity of structurally integrated visual objects, their relative positions and dynamic characteristics. Subjectively, our „visual imagery.‟

Meaning

Propositional (PROP)

A description of entities and relationships in semantic space conveying the attributes and identities of underlying referents and the nature of relationships among them. Subjectively, specific semantic relationships („knowing that‟).

Implicational (IMPLIC)

An abstract description of human existential space, abstracted over both sensory and propositional input, and conveying ideational and affective content: schematic models of experience. Subjectively, „senses‟ of knowing (e.g., „familiarity‟ or „causal relatedness‟ of ideas), or of affect (e.g., apprehension, desire).

Table 3 - The functions of the each subsystems[66]



of 47

6 Conclusion and Future Work

Aimed to be worn on-body like clothes and be always on and always accessible, the wearable computer allows information access anytime and anywhere.

The wearable computer is different from the personal assistants, mobile phone and others because it doesn't need the user intervention before use, such turning it on.

Technological progress allows make wearable computer equipment

smaller and more powerful. However, there are some challenges, like providing input and output according to privacy, low-power, unobtrusive and other requirements.

In this work we have presented a survey of the definitions, challenges

and input and output devices of wearable computers. We believe that the most important challenge is providing devices and interfaces to interact with the wearable computer. We agree that in the future, the user will carry personalized devices those can provide input or output of information. We also agree that according to the context, the user will choose the device that best fit the task‟s requirement or the user‟s preference.

Thus, we pointed out that as the users can use different devices to

interact with the computer, the software applications should work with these devices without having to be modified.

Then, we proposed to build a model of equivalence between devices in order to find a correspondence of some input devices and the mouse and the keyboard, which are the most common input devices.

Our objective with this model is improving the development of

multimodal and interactive applications for wearable computers. The expected model should provide a flexible way of changing the input device by another without having to modify the main software application.

We build our first model based on the different types of input devices

taxonomies, mainly by classifying devices that are user for text entry and pointing tasks. In this work, we have present and discussed the taxonomies of input devices and their utilization in our model in the Chapter 3.

Of course, that our first model has some issues like all the others

taxonomies presented. However, this first model allowed us to evaluate that a functional equivalence is not sufficient to our purpose. It means that two pointing devices used to draw a same geometric form can be nonequivalent considering the user‟s interaction with the devices.



of 47

Thus, we identified the importance of the human factors, such physical capabilities and cognitive process of the user, in the development of our model of equivalence.

For it, we studied some models of interaction in multimodal and

interactive systems that are present at the chapter 4. Through these models, we saw the relationship between the action and perception. Every action is preceded by an evaluation of the information captured by our senses. Then, we built a model of interaction that distinguishes the physical interaction from the cognitive process.

Distinguishing the physical interaction from the cognitive process

involved in the interaction with the device in the Chapter 5, we found that some devices that are functional equivalents are not equivalents at the physical level of interaction. We presented examples of this situation and also examples of nonequivalence at the cognitive level.

Of course that we need to study in a more detailed way the cognitive

process involved by the user during interactions with devices. A first approach consists in evaluating such process for different kinds of interactions with an experimental approach, using methods such as the Fitts‟ Law [24] or the Steering Law [1]. Another approach consists in considering human process model, such as the Interacting Cognitive Subsystem (ICS) [6] model.



of 47

References

[1] Accot J., and Zhai S., "Beyond Fitts' law: Models for trajectory-based hci tasks". In Proceeding of ACM CHI‟97 Conference on Human Factors in Computing Systems, ACM Press, pp. 295-302, 1997. Accessed: 17/06/05. Available at: http://www1.acm.org/sigs/sigchi/chi97/proceedings/paper/ja.htm [2] Akamastu M., MacKenzie I. S. and Hasbrouq T., “A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type device”. Ergonomics, Vol. 38, N° 4, pp. 816-827, 1995. Accessed: 13/06/05. Available at: http://www.yorku.ca/mack/Ergonomics.html [3] Amft O., Lauffer M., Ossevoort S., Macaluso F., Lukowicz P., and Tröster G., “Design of the QBIC wearable computing platform”. ASAP 2004: Proceedings of the 15th IEEE International Conference on Application-specific Systems, Architectures and Processors, Galveston, Texas, September 27-29, 2004. Accessed: 19/05/05. Available at: http://www.wearable.ethz.ch/fileadmin/pdf_files/pub/asap2004_qbic.pdf [4] Ark S., and Selker T., “A look at human interaction with pervasive computers”. IBM Systems Journal, Vol.38, N° 4, pp. 504-507, 1999. Accessed: 20/05/05. Available at: http://www.research.ibm.com/journal/sj/384/ark.pdf [5] Azuma R. T., "A Survey of Augmented Reality". Presence: Teleoperators and Virtual Environments Vol. 6, N° 4 pp. 355-385, August, 1997. Accessed: 04/04/05. Available at: http://www.cs.unc.edu/~azuma/ARpresence.pdf [6] Barnard P., "Interacting Cognitive Subsystems: modelling working memory phenomena within a multi-processor architecture". In A. Miyake & P. Shah (Eds), Models of Working Memory: Mechanisms of active maintenance and executive control. New York. Cambridge University Press. Chapter 9, pp. 298-339, 1999. [7] Beaudouin-Lafon M., "Interaction instrumentale : de la manipulation directe à la réalité augmentée". Actes Neuvièmes Journées sur l'Interaction Homme-Machine, IHM'97, Poitiers, Cépaduès-Editions, septembre, 1997. Accessed: 13/04/05. Available at: http://www-ihm.lri.fr/~mbl/INSTR/ihm97-paper/introduction.html [8] Beaudouin-Lafon M., "Instrumental Interaction: An Interaction Model for Designing Post-WIMP Interfaces". Proc. ACM Human Factors in Computing Systems, CHI 2000. The Hague (Netherlands), ACM Press, pp 446-453, 1-6 April, 2000. Accessed: 01/06/05. Available at: http://www-ihm.lri.fr/~mbl/papers/CHI2000/paper.pdf



of 47

[9] Billinghurst M., Starner T., “Wearable Devices: New Ways to Manage Information”, IEEE Computer, Vol. 32, No. 1, pp. 57-64, January, 1999. Accessed: 10/03/05. Available at: http://www.cs.utexas.edu/users/dburger/teaching/cs395t-s99/papers/25_wearable.pdf [10] Blasko G., and FEINER S.,"A Menu Interface for Wearable Computing", 6th International Symposium on Wearable Computers (ISWC 2002), pp. 164–165, 2002. Accessed: 22/04/05. Available at: http://www1.cs.columbia.edu/~gblasko/Publications/Blasko_ISWC02.pdf [11] Bleul S., Schaefer R., and Mueller W., “Multimodal Dialog Description for Mobile Devices”: Workshop on XML-based User Interface Description Languages at AVI 2004, Gallipoli, Italy, May, 25, 2004. Accessed: 10/03/05. Available at: http://www.c-lab.de/~wolfgang/avi04.pdf [12] Bodine K., and Gemperle F., "Effects of Functionality on Perceived Comfort of Wearables". Proceedings of the Seventh IEEE International Symposium on Wearable Computers. White Plains, NY, October, pp. 57-60, 2003. Accessed: 11/03/05. Available at: http://www.lerru.com/portfolio/research/functioncomfort/bodine-iswc03.pdf [13] Booth S., and Schmidt-Tjarksen T., "Psychological Theory in Haptic Interface Design: initial steps towards an Interacting Cognitive Subsystems (ICS) approach. EuroHaptics, 2001. Accessed: 13/06/05. Available at: http://www.eurohaptics.vision.ee.ethz.ch/2001/booth.pdf [14] Bowman D., Kruijff E., LaViola J. Jr., and Poupyrev I., 3D User Interfaces: Theory and Practice. Addison-Wesley, Boston, 2004. [15] Brewster S. A., "Multimodal interaction and proactive computing". In British Council workshop on Proactive Computing, Nizhny Novgorod, Russia, 2005. Accessed: 17/06/05. Available at: http://www.dcs.gla.ac.uk/~stephen/papers/proactive_computing_paper.pdf [16] Buxton W., “Lexical and Pragmatic Considerations of Input Structures”. Computer Graphics, Vol. 17, N° 1, pp. 31-37, 1983. Accessed: 31/05/05. Available at: http://www.billbuxton.com/lexical.html [17] Buxton W., Billinghurst M., Guiard Y., Sellen A. and Zhai S., “Human Input to Computer Systems: Theories, Techniques and Technology”, DRAFT, 1994 – 2002. Accessed: 18/05/05. Available at: http://www.billbuxton.com/inputManuscript.html



of 47

[18] Card S. K., Mackinlay J. D., and Robertson G. G., “A Morphological Analysis of the Design Space of Input Devices”. ACM Transactions on Information Systems Vol. 9, N° 2, pp. 99-122, 1991. Accessed: 11/05/05. Available at: http://www2.parc.com/istl/projects/uir/pubs/items/UIR-1991-02-Card-TOIS-Morphological.pdf [19] Crabtree I. B., Rhodes B. J., “Wearable Computing and the Remembrance Agent”, BT Technology Journal, Vol. 16, No. 3, pp. 118-124, July, 1998. Accessed: 26/05/05. Available at: http://alumni.media.mit.edu/~rhodes/Papers/bt-journal-wearables.html [20] Clarke K. C., Nuernberger A., Pingel T., and Qingyun D., “User Interface Design for a Wearable Field Computer”. In Proc. of National Conference on Digital Government Research, Los Angeles, CA, 2002. Accessed: 04/04/05. Available at: http://dg.statlab.iastate.edu/dg/papers_presentations/pdfs/clarke_et_al_draft_dgo_02.pdf [21] Dragicevic P., “Un modèle de configurations d'entrée pour des systèmes interactifs multi-dispositifs hautement configurables“, PhD Thesis, march 2004. Accessed: 11/02/05. Available at: http://liihs.irit.fr/dragice/these/ [22] Enderle G., Kansy K., and Pfaff G., “GKS - the graphics standard”, Computer Graphics Programming, New York, 1984. [23] Espoo, VTT Electronics, “Methods and technologies for experimenting with ubiquitous computing”. VTT Publications, 2005. Accessed: 25/05/05. Available at: Available at http://www.vtt.fi/inf/pdf/publications/2005/P560.pdf [24] Fitt P.M., "The information capacity of the human motor system in controlling the amplitude of movement". Journal of Experimental Psychology, vol 47, no 6, pp. 381-391, 1954. [25] Foley J., and Wallace V., “The art of natural graphics man-machine conversation”. Proceedings of the IEEE, Vol. 62, n° 4, pp. 462-471, 1974. [26] Foley J.D., Wallace V.L., and Chan P., “The human factors of computer graphics interaction techniques”. IEEE Computer Graphics and Applications, Vol. 4, N° 11, pp. 13-48, November, 1984. [27] Friedemann M., "Wireless Future: Ubiquitous Computing", Proc. of Wireless Congress 2004, Munich, Germany, November, 2004. Accessed: 26/02/05. Available at: http://www.vs.inf.ethz.ch/publ/papers/mattern2004_electronica.pdf



of 47

[28] Fukumoto M., and Sugimura T., "Fulltime-wear Interface Technology". New Interface Technologies Creating Innovations in Mobile Communication. Vol.1, N° 8, November, 2003. Accessed: 16/03/05. Available at: http://www.ntt.co.jp/tr/0311/files/ntr0411077.pdf [29] Gabbard J. L., and Hix D., “A taxonomy of usability characteristics in virtual environments”, Office of Naval Research, November, 1997. Accessed: 01/06/05. Available at: http://www.movesinstitute.org/Theses/Tokgoz_thesisWeb/DataBase/gabbard_hix_taxonomy.pdf [30] Goldberg D., and Richardson C., "Touch-typing with a stylus". Proceedings of the ACM Conference on Human Factors in Computing Systems - INTERCHI '93, pp. 80-87. New York: ACM, 1993. [31] Hinckley K., Jacob R.J.K., Ware and C., “Input/output Devices and Interaction Techniques”. In The Computer Science Handbook, Second Edition, ed. by A.B. Tucker, pp. 20.1-20.32, Chapman & Hall/CRC Press, 2004. Accessed: 01/03/05. Available at: http://www.cs.tufts.edu/~jacob/papers/crc2.pdf [32] Jacob R.J.K., Leggett J.J., Myers B.A., and Pausch R., "Interaction Styles and Input/Output Devices". Behavior and Information Technology, vol. 12, n° 2, pp. 69-79, 1993. Accessed: 09/03/05. Available at: http://www.cs.tufts.edu/~jacob/papers/bit.pdf [33] Jacob R.J.K., Sibert L.E., McFarlane D.C., and Mullen M.P.Jr., “Integrality and Separability of Input Devices”. ACM Transactions on Computer-Human Interaction, Vol. 1, n° 1, pp. 3-26, March, 1994. Accessed: 20/05/05. Available at: http://www.cs.tufts.edu/~jacob/papers/tochi.pdf [34] Locher I., Junker H., Kirstein T., and Tröster G., “Wireless, Low-Cost Interface for Body Area Networks”, Proc. of 8th IEEE International Symposium on Wearable Computers, Arlington VA, pp. 170-17, October - November, 2004. Accessed: 10/03/05. Available at: http://csdl2.computer.org/comp/proceedings/iswc/2004/2186/00/21860170.pdf [35] Kabbash P., MacKenzie I.S., and Buxton W., "Human performance using computer input devices in the preferred and non-preferred hands". Proceedings of InterCHI '93, pp. 474-481, 1993. Accessed: 29/06/05. Available at: http://www.billbuxton.com/LHfitts.html



of 47

[36] Kölsch M., and Turk M., "Keyboards without keyboards: a survey of virtual keyboards", Workshop on Sensing and Input for Media-centric Systems, Santa Barbara, CA, June 20-21, 2002. Accessed: 04/03/05. Available at: http://ilab.cs.ucsb.edu/projects/mathias/KolschKeyboards.pdf [37] Lyons K., Plaisted D., and Starner T., "Expert Chording Text Entry on the Twiddler One-Handed Keyboard". Georgia Institute of Technology, 2004. Accessed: 09/03/05. Available at: http://smartech.gatech.edu:8282/dspace/bitstream/1853/58/1/04-09.pdf [38] Mackinlay J. D., Card S. K., and Robertson G. G., “A Semantic Analysis of the Design Space of Input Devices”. Human-Computer Interaction, Vol. 5(2-3), pp. 145-190, 1990. Accessed: 11/05/05. Available at: http://www2.parc.com/istl/projects/uir/pubs/items/UIR-1990-05-Mackinlay-HCI-SemanticDesignSpace.pdf [39] Mann S., "Wearable Computing: A First Step toward Personal Imaging", Computer, Vol. 30, No. 2, February, 1997. Accessed: 21/03/05. Available at: [40] Mann S., “Wearable Computing as Means for Personal Empowerment”. Keynote Address for The First International Conference on Wearable Computing, ICWC'98, May 1998. Accessed: 21/03/05. Available at: http://wearcam.org/icwckeynote.html [41] Matias E., MacKenzie I.S., and Buxton W., “One-Handed Touch-Typing on a QWERTY Keyboard”. In Proceedings of Human-Computer Interaction, Vol. 11, pp. 1-27, 1996. Accessed: 08/04/05. Available at: http://edgarmatias.com/papers/hci96/ [42] Milanski J., "Displacing the Keyboard: Voice recognition takes off".ID522 Technological Development and Design Innovation. Spring 1998. Accessed: 07/04/05. Available at: http://www.id.iit.edu/~milanski/strategy/speech/speech.pdf [43] Moeslund T. and Nørgaard L., “A brief overview of hand gestures used in wearable human computer interfaces”. Laboratory of Computer Vision and Media Technology, Aalborg University. Technical report: CVMT 03-02, ISSN: 1601-3646, 2003. Accessed: 25/05/05. Available at: http://www.cvmt.dk/~tbm/Publications/gesture-hci.pdf [44] Mueller W., Schaefer R., and Bleul S., “Interactive Multimodal User Interfaces for Mobile Devices”. In Proceedings of the Proceedings of the 37th Annual Hawaii international Conference on System Sciences (Hicss'04) - Track 9 - Vol. 9, HICSS. IEEE Computer Society, Washington, DC, January 05 - 08, 2004.



of 47

Accessed: 09/03/05. Available at: http://csdl2.computer.org/comp/proceedings/hicss/2004/2056/09/205690286a.pdf [45] Newman N. J., “A Wearable Application Integration Framework”, University of Essex, 2000. Accessed: 11/02/05. Available at: http://www.cs.washington.edu/sewpc/papers/newman.pdf [46] Nilsson M., Drugge M., and Parnes P., "In the borderland between wearable computers and pervasive computing". Research report, Luleå University of Technology. 2003. Accessed: 16/02/05. Available at: http://epubl.luth.se/1402-1528/2003/04/LTU-FR-0304-SE.pdf [47] Orio N., Schnell N., and Wanderley M. M., "Input Devices for Musical Expression: Borrowing Tools from HCI". New Interfaces for Musical Expression Workshop (NINE-01), ACM CHI'01, Seatle, USA, April, 2001. Accessed: 01/06/05. Available at: http://ame2.asu.edu/faculty/dab/classes/interactiveTechFall04/papers/orio.pdf [48] Perng J. K., Fisher B., Hollar S., and Pister K. S. J., "Acceleration Sensing Glove". In Proceedings of the 3rd IEEE International Symposium on Wearable Computers, ISWC '99. pp. 178, 1999. Accessed: 31/03/05. Available at: http://www-bsac.eecs.berkeley.edu/archive/users/hollar-seth/publications/iswc.pdf [49] Rhodes B. J., “The Wearable Remembrance Agent. A system for augmented memory”. Proceedings of the International Symposium on Wearable Computing, IEEE, pp. 123-128, October 1997. Accessed: 26/02/05. Available at: http://www.bradleyrhodes.com/Papers/wear-ra.html [50] Sawhney N., and Schmandt C., "Speaking and Listening on the Run: Design for Wearable Audio Computing". Proceedings of the International Symposium on Wearable Computing, October, 1998. Accessed: 04/04/05. Available at: http://c2000.cc.gatech.edu/classes/cs8113c_99_spring/readings/sawhney.pdf [51] Schmidt A., Gellersen Hans-W., Merz C., “Enabling Implicit Human Computer Interaction - A Wearable RFID-Tag Reader”, Proc. of the International Symposium on Wearable Computers, ISWC2000, pp. 193-194, Atlanta, GA, USA, October, 2000. Accessed: 16/02/05. Available at: http://www.comp.lancs.ac.uk/~albrecht/pubs/pdf/schmidt_iswc2000_wearable-rfid-tag-reader.pdf [52] Schmidt A., Gellersen H.-W., Beigl M., and Thate O., "Developing User Interfaces for Wearable Computers: Don‟t Stop to Point and Click".



of 47

International Workshop on Interactive Applications of Mobile Computing (IMC2000), pp. 9-10. November, Warnemünde. Institut für Grafische Datenverarbeitung, Fraunhofer Gesellschaft, 2000. Accessed: 16/02/05. Available at: [53] Schomaker L., Nijtmans J., Camurri A., Lavagetto F., Morasso P., Benoit C., Guiard-Marigny T., Le Goff B., Robert-Ribes J., Adjoudani A., Defee I., Munch S., Hartung K., and Blauert J., “A Taxonomy of Multimodal Interaction in the Human Information Processing System”. Multimodal Integration for Advanced Multimedia Interfaces (MIAMI). ESPRIT III, Basic Research Project 8579. 1995. Accessed: 09/03/05. Available at: http://hwr.nici.kun.nl/~miami/ [54] Sebe N., Lew M.S. and Huang T.S., “The State-of-the-Art in Human-Computer Interaction”, International Workshop on Human Computer Interaction (HCI'04 Proceedings), pp. 1-5, Prague, Czech Republic, May 2004. Accessed: 25/02/05. Available at: http://staff.science.uva.nl/~nicu/publications/hci04_edit.pdf [55] Soukoreff R. W., and MacKenzie I. S., "Theoretical upper and lower bounds on typing speed using a stylus and soft keyboard". Behaviour & Information Technology, Vol. 14, pp. 370-379, 1995. Accessed: 25/05/05. Available at: http://www.yorku.ca/mack/bit95.html [56] Stäger M., Lukowicz P., Perera N., Buren, T. von, Troster G., Starner T., "SoundButton: Design of a Low Power Wearable Audio Classification System".Proceedings of the 7th International Symposium on Wearable Computers, pp. 12-17, New York, USA, 21-23 October, 2003. Accessed: 09/03/05. Available at: http://www.wearable.ethz.ch/fileadmin/pdf_files/pub/staeger_iswc03.pdf [57] Starner T., "The Cyborgs are Coming, or The Real Personal Computers". Unpublished paper. 1994 Accessed: 01/04/05. Available at: http://vismod.media.mit.edu/pub/tech-reports/TR-318-ABSTRACT.html [58] Starner T., and Maguire Y., “Heat dissipation in wearable computers aided by thermal coupling with the user”. Mobile Networks and Applications, 4, pp.3-13, 1999. Accessed: 27/05/05. Available at: http://www.media.mit.edu/physics/publications/papers/99.01.MONET.pdf [59] Starner T., “The challenges of wearable computing: Part 1”. IEEE Micro, Vol. 21, N° 4, pp.44–52, July-august 2001. Accessed: 11/02/05. Available at: http://www.ece.umd.edu/courses/enee759m.S2002/papers/starner2001a-micro21-4.pdf Felix Garcia and Javier Fernandez, POSIX Thread Libraries.



of 47

[60] Starner T., “The challenges of wearable computing: Part 2”. IEEE Micro, Vol. 21, N° 4, pp.54–67, July-august 2001. Accessed: 11/02/05. Available at: http://www.ece.umd.edu/courses/enee759m.S2002/papers/starner2001b-micro21-4.pdf [61] Starner T., "The role of speech input in wearable computing". IEEE Pervasive Computing, Vol. 1, n° 3, pp. 89-93, 2002. Accessed: 21/03/05. Available at: http://www.ee.oulu.fi/~skidi/teaching/mobile_and_ubiquitous_multimedia_2002/the_role_of_speech_input.pdf [62] Tsegaye M., "Wearable Computing: The future of computing". Technical Report, Computer Science Department, Rhodes University, 2003. Accessed: 25/05/05. Available at: http://www.cs.ru.ac.za/research/students/g98t4414/static/papers/wearablehs.pdf [63] Turk, M., and Kölsch M., "Perceptual interfaces". In Gerald Medioni and Sing Bing Kang, editors, Emerging Topics in Computer Vision, chapter 10. Prentice Hall. 2004. Accessed: 10/06/05. Available at: http://ilab.cs.ucsb.edu/projects/turk/TurkKolsch%202004.pdf [64] Weiser M., The Computer for the 21st Century, Scientific American, Vol. 265, N° 3, pp.66–75, September 1991. Accessed: 28/01/05. Available at: http://www.ubiq.com/hypertext/weiser/SciAmDraft3.html [65] Wöhler J., "Driver Development for TouchGlove Input Device for DWARF based Applications", Project report. Technische Universität München, 2003. Accessed: 22/04/05. Available at: http://wwwnavab.in.tum.de/twiki/pub/DWARF/SepWoehler/SEP-TouchGlove.pdf Web Links [66] Cognition and Brain Sciences Unit. Available at: http://www.mrc-cbu.cam.ac.uk/External/index.shtml [67] FrogPad™, Inc. Available at: http://www.frogpad.com/ [68] Handykey Corporation. Available at: http://www.handykey.com [69] L3 Systems. Available at: http://www.l3sys.com



of 47

[70] Microsoft Corporation. Available at: http://www.microsoft.com [71] Matias Corporation. Available at: http://halfkeyboard.com/halfkeyboard/index.html [72] MicroOptical Corporation. Available at: http://www.microopticalcorp.com/ [73] Palm, Inc. Available at : http://www.palmone.com/ To Appear [74] de Morais W. O., Tigli J.-Y., Lingrand D., "Dynamic Management of Input/Output Devices for Wearable Computers", to appear in Wearable Futures, University of Wales, Newport, WALES, UK, 14 - 16 September, 2005. [75] Lingrand D., de Morais W. O., Tigli J.-Y., "Ordinateur porté: gestion des dispositifs d'entrée". To appear in 17° Conférence Francophone sur lnteraction Homme-Machine. IHM'05, Toulouse, 27-30 septembre, 2005.

Dynamic Management of Input/Output Devices for Wearable ...589182/FULLTEXT01.pdfDe Morais W.O....

Documents

Transcript of Dynamic Management of Input/Output Devices for Wearable ...589182/FULLTEXT01.pdfDe Morais W.O....