Master thesis: Smartphone Controlled Home Environments

SMARTPHONE CONTROLLEDHOME ENVIRONMENTS

Suhwa Lee

MSc Design for InteractionFaculty of Industrial Design Engineering Delft University of Technology

2

STUDENT Suhwa Lee MSc Design for Interaction Faculty of Industrial Design Engineering Delft University of Technology

CHAIR Dr. Ir. G.J. Pasman Associate Professor Department of Industrial Design

MENTOR Dr. Ir. A.P.O.S. Vermeeren Assistant Professor Department of Industrial Design

A thesis submitted in partial fulfilment of the requirements for the degree of Design for Interaction at the Faculty of Industrial Design Engineering

March 2016 Delft, the Netherlands

3

SMARTPHONE CONTROLLED HOME ENVIRONMENTS

ACKNOWLEDGEMENT

PREFACEThis thesis started from an imagination of performing choreography to control appliances at home. I always had my interest in realizing more playful physical interactions, instead of immersing into the digital world with touch screens. And I found a renown design agency who can support the realization of my imagination through a form of the master thesis project, whose focus was on expanding their knowledge in smart living.

The researches figured out the playfulness that I had in mind was not a significant contributor to the smart living. Functional requirements are seen more important in smart living products which were due to the users who pursues them. While trying to connect smart living and gesture control together, I was able to gain a precious insight. The future of smart living or, at least, the current expectation for the smart living is to automate and be intelligent, and this cannot coexist with controlling, which definitely requires user inputs. In the end, other solutions for compromising between two worlds came along, and I am truly thankful that I am now able to finish my master study.

Throughout the process, there were many changes or building prototypes to simulate the effects of the concept, and this was very joyful. This master thesis not only provided me chances to utilize my knowledge accumulated through studying and working in the design field for the last seven years, but also taught me new research, design, and making skill sets along with crisis management skills.

I hope that this thesis contributes and raises interesting questions in the field of gesture control. Furthermore, as a student who have pursued master’s degree in the Design for Interaction, I hope the future of smart living can develop in a more human-centered way. Appreciating higher values than just mere convenience and comfort, and making the world a better place as one of my interview participants mentioned during the interview.

I hope the world is awaiting with a lot of fun projects. World, please welcome Suhwa Lee, after 7 years of studies!

Working on this master thesis away from home and even away from the Netherlands, which now feels like my second home country, was a big challenge. I would very much like to thank my chair and mentor, Gert Pasman and Arnold Vermeeren for their support from distance, and in Delft as well. Without their timely help to coordinate where I have to head next, it would not have been possible for me to start writing this piece. I would also like to thank my company mentor, Damion Bailey, for giving me a chance to commence a thesis project at Designit, and for being a constant inspiration.

I would also like to show my appreciation for the research participants who were willing to spare time and give valuable feedback which also made this thesis possible.

Last but not the least, I would like to thank my friends and my family. I am grateful for your hands, emotional supports and laughter given when I needed throughout writing this thesis and my life in general.

Suhwa Lee

March 2016

5

TABLE OF CONTENTS

INTRODUCTIONBackground

Assignment

RESEARCHDesk Research

User Research

Conclusion

IDEA GENERATIONIdeation 1: Gesture-triggered Home environments

Ideation 2: Benefits of Gestures

Concept Generation

Evaluation of the Concepts

Preliminary Concept: Smartphone Gesture Control

CONCEPT DEVELOPMENTIteration 1: Smartphone Function-Gesture Mapping

Iteration 2: Wireframe Concept Development

Iteration 3: Smartphone Remote Control

FINAL DESIGNConcept Description

Features

User Scenarios

Technology Research

EVALUATIONSetup

Results and Analysis

CONCLUSION

REFERENCES

APPENDIX

1. 1.1.

1.2.

2. 2.1.

2.2.

2.3.

3.3.1.

3.2.

3.3.

3.4.

3.5.

4. 4.1.

4.2.

4.3.

5. 5.1.

5.2.

5.3.

5.4.

6. 6.1.

6.2.

7.

8.

9 10

12

1718

26

31

3334

36

38

40

46

4950

58

70

8182

84

94

96

99100

102

107

111

115

SMART PHONE CONTROLLED HOME ENVIRONMENTS

7

SMART PHONE CONTROLLED HOME ENVIRONMENTS

EXECUTIVE SUMMARY

This Master thesis “Smartphone controlled home environments” was initiated to seek integration of Gesture control into a Smart living context.

This project goes through five different phases to achieve the objectives; Research, Idea generation, Concept development, Final design and evaluation.

The desk research and user research lead to two significant insights: Gestures demanding less mental effort (intuitive) has the potential to be ‘smart’, and intelligence, automation through the prediction of user needs, is the future of Smart living.

The ideation phase covers the exploration of two potential design directions. The first direction leads to a significant finding of this thesis: the intelligence and Gesture control, in essence, are not compatible. The intelligence avoids any user input, and the Gesture control requires a user to make an input. The second direction leads to the idea generation using benefits of gesturing. The initial concept turns out to be Smartphone Gesture control, which uses original actions as gestural input to trigger smartphone functions.

The initial concept is developed in an iterative manner. A preliminarily defined gesture library gets refined through the first user test. In the second iteration, the concept splits into three sub-functions considering the interaction scenario: gesturing, gesture recognition, and application trigger. The solutions are again evaluated with the users. The remote control interaction is designed further in the third iteration, and the concept becomes more relevant to Smart living and home environments.

The final design presents a smartphone gesture library which triggers twelve functions allowing the control of the smartphone and home environment. The functions include; ‘Wake up the device’, ‘Make a call’, ‘Answer a call’, ‘Snooze a call’, ‘Reject a call’, ‘Launch the messenger’, ‘Activate the camera’, ‘Launch the recorder’, ‘Snooze an alarm’,’ Turn off an alarm’,’ Launch the music app’, and ‘Start remote control’. The ‘Start remote control’ function allows the home appliance control which applies to Blinds/curtains, lighting, speaker, and thermostat. Most of the gestures are derived from original actions, thus, it saves physical and mental efforts of the user, and feels smart.

The final evaluation study assessed the final design. The concept is considered intuitive, easy to use, and smart. The main reason for the smartness is: Effort-saving control gives a sense of automation which applies to the home environment. More study on fatigue and interaction speed mapping is recommended to contribute better user experience.

In conclusion, this master thesis, “Smartphone controlled home environments” achieved an interaction design that shapes user’s life smarter, by providing intuitive and effortless Gesture control realized through a smartphone that applies to both of the smartphone and home environments.

INTRODUCTIONThis thesis is about introducing new gesture interaction concept “Smartphone controlled home environments”. This chapter starts with providing related background knowledge regarding the development of user interface, Gesture control, and Smart living, as a context of the design. It also offers a short introduction of how the use of smartphone relates to this thesis. It ends with the details of the assignment pursued through the thesis.

1

CHAPTER 1.

10

1.1.BACKGROUND

10

1.1

USER INTERFACES

User interface nowadays has become one of the main interests in shaping a better product experience. Beyond meeting its functional requirements, well-designed interface contributes to a seamless experience between the user and product (Rouse, 2005; Desmet & Hekkert, 2007).

In the past, the design of user interfaces used to be very dependent on available technology (“The Best Computer Interfaces: Past, Present, and Future | MIT Technology Review”, 2009; Mayhew, 2008; Myers, Hudson & Pausch, 2000). However, many changes have been made in the field of User interface design ever since, in order to meet more complex requirements (Figure 1-1).

Old hardware interfaces (e.g. industrial control panel, physical remote control, mouse) are being replaced by more advanced alternatives such as Graphical User Interface (GUI), Natural User Interface (NUI), and Brain Computer Interface (BCI) (Grudin, 2005; MIT Technology Review, 2016). Technological development boosted such a phenomenon as well (“Revolutionary User Interfaces”, 2015). Among all these new alternatives, Natural User Interface (NUI) provides the highest level of intuitiveness to the end users. Natural User Interface works as an extension of a human body. It removes metaphor or any middleware and eliminates the physical and psychological distance between human and machine (Hornecker & Buur, 2006; Mann, 2002).

GESTURE CONTROL

The development of Natural User Interface opened up new possibilities. For instance, it allowed devices to accommodate to natural human ability as a controlling method, such as voice, body gestures and even eye movements. Good examples would be Siri, Kinect, Smart TVs, Google glass, etc. However, these devices have shown limited usage (Yan & Aimaiti, 2011; Koster-j, 2006). The lack of objects to control seems to contribute to the limited usage. For example, Siri is solely for controlling iPhone. Kinect serves as an input device mainly for a TV or a computer in the field of entertainment.

Removing any third-party device and use one’s own ability to control products around us seems to be a fascinating idea. It is something more close to the intuitiveness that we always pursue, not having to learn how to use a foreign device. It also eliminates the chance of misplacing the foreign device as well.

The focus of this thesis is to increase the acceptance of Gesture control at home environments. Among different types of gestures, the bodily gesture was set to be the primary focus of this thesis, while not entirely excluding the collaboration with other methods such as voice. The main reason behind this was that utilizing the whole body seemed to have more space for the creativity and be potential to be integrated with home environments. Voice Control was seen to have much less opportunity for design interventions. It is strongly dependent on a language and the voice recognition technology itself, the right phrase has to be said then recognized by the system correctly. Eye tracking did not seem promising either. Compared to bodily gestures, eye movements only have a very limited language library. Hence, the technology is suitable for few cases where tracking itself is the core task. Thus, it was difficult to see a potential to have extensive control over living space.

INTRODUCTION

11


11

SMART LIVING

Smart living concepts have arisen due to many technological innovations, and it has benefited human living (Harper, 2003). Smart living can be applied to various domains, such as promoting comfort, convenience, healthcare, entertainment, communication, etc. The primary focus of this thesis is to enhance ‘control’ in the smart home using Gesture control to achieve a better daily life experience. If there is a mass adoption of Gesture control related to Smart living, it will need to consider its users and the appropriate contexts of use. The expectation is that it will bring various positive influences, especially concerning the enhancement of user experience in the home.

As our living environments change, new products are being designed to meet the emerging needs. The way we control and interact with products is changing as well, interfaces between human and product have been modified radically over a rather short time to be more user-friendly. Recently, devices that enable Gesture control have been developed (e.g. Samsung Smart TV, ThalmicLab Myo band, etc.), which minimize and almost eliminate the need for having a physical interface between human and product at all. However, current Gesture control devices on the market focus on controlling smart devices, while lacking a discussion about the benefits of introducing Gesture control in a holistic context.

SMARTPHONE

The use of smartphone significantly links to the final design of this thesis. A short introductory background regarding the link between this thesis and the use of a smartphone is briefly explained as below.

A smartphone is a mobile device that we use on a daily basis. It provides various functions that benefit its users. The smartphone was not given as a context for this thesis in the first place. The idea of utilizing a smartphone occurred during the ideation phase. Smartphone roles as both a medium and the target of Gesture control in the final design outcome of this thesis. It also links Gesture control and the Smart living context.

Figure 1-1. A brief history of of User Interfaces: Remote control, Mouse, Touchpad, Multitouch, Kinect (NUI using Gestures), Siri (NUI using voice), Google glass (NUI), and Brain-User-Interface (BUI)

CHAPTER 1.

12

PROBLEM DEFINITION

Gesture control is gaining more interest, and becoming a natural way to interact with products. However, the application of Gesture control still seems to be restricted. One case is a single product that is controllable with gestures (e.g. Samsung Smart TV, Xbox and Nintendo Wii); another is a Gesture control enabler used for very specific devices (e.g. Leap Motion, Kinect, and MYO band) rather than a control in a holistic context such as a home environment. Considering the convenience that Gesture control can bring, its expansion to a whole environmental context seems to be natural. A successful interaction of gestures control and daily life requires a thorough and grounded understanding of both users and the context.

Especially, a Smart living context gives a reason to explore user experience, focusing on values and emotions over a mere technological implementation. Previous research indicates that people are still confused about how to react to the new technology and that there are contradicting opinions toward Smart living. While some people show enthusiasm to adopt new technology to shape their lives more conveniently and smart, others seem reserved and show worries of mechanizing or dehumanizing their lives (Harper, 2003). Through the project, user needs and how Gesture control can benefit them will be addressed.

INVOLVED PARTIES

TU Delft IDEThis project pursues to come up with a product-service system that can be implemented in a consumer market, thus fitting the domain of Industrial Design Engineering. Various tools and techniques learned from the Design for Interaction Master program are to be utilized actively throughout the execution of the project.

The mission of TU Delft IDE faculty is to contribute to the development of products or services for people on the basis of the balanced interests of users, industry, society, and the environment. As such the project directly contributes to this mission. It will contribute to technological advancements in a sense that it will develop appropriate ways to adopt technology close to human life, using a human-centered approach.

This project will also extend scientific knowledge by providing new design insights in the context of Smart living, which is one of the rising fields in HCI (Human Computer Interaction) for the better future. It will explore how available gesture recognition technology can be better shaped to be more beneficial to its end-users through a design research/project.

DesignitDesignit is a global strategic design firm, specialized in designing product-service experiences, making business transformation happen. Smart living is one of the three major themes that the company is focusing on for the future. They aim to design human-centered and technology-infused solutions that enable humans to live easier and better lives through intelligent products and services. This project, introducing Gesture control to home environments will contribute to the company’s knowledge in the Smart living.

ASSIGNMENT1.2.

Figure 1-2. Parties involved in this thesis

INTRODUCTION

13


SOLUTION PROPOSITION

The company seeks to establish a Smart living concept by design. The home environment is where we get to spend a lot of time during our lives. Various product experiences take place at home, which significantly contributes to a lifestyle. Thus, the expected outcome of this project, to introduce Gesture control in a Smart living context based on available gesture recognition technology highly fits the company’s interest.

In short, the assignment is to design a Gesture control interaction concept that can be successfully integrated into the Smart living environment, by conducting research on the context of when, where and what is being controlled and recognized affordances of a given task. It aims to flatten a learning curve thus enhancing the overall user experience.

THESIS STRUCTURE

This thesis goes through five different phases; Research, Idea generation, Concept development, Final design, and Evaluation. These main phases constructs the framework of the thesis.

The first chapter introduces the background information of this thesis. The initiation, background knowledge, involved parties, and solution propositions are covered.

Chapter 2 covers the research phase. In the research phase, a desk research covered gestures, gesture control, and smart living. User interviews were done to collected more user-centered insights from the real users. The collected information was translated into meaningful insights for the next phase, such as potential opportunities for the gesture control.

Chapter 3, Idea generation, covers the exploration of two different directions. Among the two, the first ideation, Gesture triggered home automation, was withdrawn in the end thus it is only shortly covered in this chapter. It leads to the introduction of the second ideation, Benefits of using gestures. This further develops into concept generation and evaluation of the primitive, initial concepts. At the end of the chapter, one concept is selected to proceed.

Chapter 4, Concept development, demonstrates an iterative process of the concept development. Three iterations are introduced; Smartphone function-gesture mapping, Interaction scenario, and Smart phone remote control. The first iteration illustrates the design process of a preliminary gesture set in a relation to smartphone functions, and a user test for the verification of the preliminarily selected gestures.

The second iteration covers the interaction design evolved from the preliminary gesture sets. The second iteration covers the development of wireframe concepts and its evaluation with the users. The third iteration includes the further development of one of the twelve functions: remote control. It demonstrates an interaction design and the design thinking process for the remote control of home appliances.

The final design is introduced in Chapter 5. The concept description, features, and user scenarios are presented. This chapter includes technology research to reinforce the feasibility of the design.

The final design is assessed through the final evaluation study with users, which are described in detail in Chapter 6. The study setup, related studies contributed to the setup are illustrated in details along with the results, analysis, and discussion for the conclusion.

In Chapter 7, this thesis closes with the conclusion and recommendations for the design. The overview of the thesis structure is provided in the next page (Figure 1-3).

CHAPTER 1.

14

INTRODUCTION

The thesis consists of seven main chapters along with references and appendices at the end. The chapters include; Introduction, Research, Idea generation, Concept development, Final design, Evaluation, and conclusion.

The thesis structure is illustrated below for the overview.

THESIS STRUCTURE

Figure 1-2. Overview of the thesis structure

15


Figure 1-3. An overview of the thesis structure

RESEARCHThis research phase covers the desk research which was conducted to obtain the required background information. It explores gestures as a communication method, along with the obvious topics such as Gesture control and smart living.

After the desk research, user interviews reveals findings of the potential users and the context. The results from both the desk and user research lead to insights for future design directions.

2

18

2.1.DESK RESEARCH

Through this research phase, desk research was conducted to obtain background information. Gesture as a communication method was also research on the top of the Gesture control and Smart living, to understand the property of the gestures. User interviews were done after the desk research to understand potential users and the context. The results of both researches led to insights into future design directions.

2.1.1 GESTURE AND COMMUNICATION

DefinitionThe Oxford dictionary defines gesture as a movement of part of the body, especially a hand or the head, to express an idea or meaning (“gesture”, n.d.). When its expressiveness of an idea or meaning is looked into, it is defined as a form of non-verbal communication (Kendon, 2004). It not only conveys messages to listeners but also help the one who makes a gesture to think better. Gesturing is claimed to have a benefit to reduce mental effort and help users to perform a task better (Klemmer, Hartmann & Takayama, 2006). A short research about the definition of gesture leads to a further research on gesture as a communication method, and how it benefits people in the given context.

Typology of GestureWhen it comes to gesture as a communication method, gestures can be classified into different types based on the various properties. According to Kendon’s Continuum (McNeill, 2000), there are iconic gestures, deictic gestures, emblematic gesture, and sign language, which are classified based on its linguistic properties. Figure 2-1 was created to help the understanding of the nature of different types of gestures.

Used with a verbal language, iconic gestures illustrate what is

being said. They assist in describing physical, and tangible items. The example would be gesturing to explain size, or shape, etc. better. Iconic gestures help to create the mental image by adding such details. Not only consciously, but they can also be used unconsciously during a speech. The deictic gesture is used to point or indicate.

Iconic gesture and deictic gesture are used by very young children as well. Babies who cannot speak yet or whose vocabulary lacks significantly use gestures as an alternative, to communicate with their parents or other people (“Child using Gestures to communicate”, 2016; Özçalışkan & Goldin-Meadow, 2005; Capone, & McGregor, 2004; Thal & Bates, 1988). Gesture language library invented for such babies can be found as well (Baby Sign Language, 2016; “Baby Sign Language”, 2016; Briant, 2004).

Emblematic gestures convey specific meanings which are often abstract (e.g. V for victory, peace, etc.) They are used consciously and understood consciously. They can be used without using a word, as they can be a substitute. The meaning of gestures is not always universal, as it can convey different meaning across the culture.

Lastly, sign language consists of gestures only and yet serves as a complete alternative to a verbal language, especially for those who cannot utilize verbal communication. Different from the gesture types as mentioned earlier, it is a complete communication itself providing extensive vocabularies to serve its purpose. The gestures are performed very conscious, requiring more mental effort of a performer.

According to Kendon (1980), gesticulation is also considered, as a more primitive level of non-verbal communication. As the type of gesture changes from gesticulation, iconic gesture, deictic gesture, emblematic gesture, to sign language, their properties become more close to a verbal language, and it can convey more sophisticated meanings. However, it also requires the speaker (gesture performer) to be more conscious, and more mental effort is required.

Figure 2-1. Typology of gestures

RESEARCHCHAPTER 2.

19


2.1.2 GESTURE CONTROL

Gesture controlBy definition, Gesture control is the ability to recognize and interpret movements of the human body in order to interact with and control a computer system without direct physical contact. The term “natural user interface” is becoming commonly used to describe these interface systems, reflecting the general lack of any intermediate devices between the user and the system (“Natural user interface”, n.d.). As such, a Gesture control device works as an extension of a human body, utilizing one’s original ability or body as an input. This removes the existence of any third-party device between the interaction of a user and an object to control. The use of third-party device, such as a remote controller has its own hardware and this can cause restriction in the interaction between the user and product as it only allows interactions to happen in a designated way in which it is designed. The use of Gesture control can make a direction connection between the user and product and is therefore expected to create a more natural interaction model (Mann, 2002). Figure 2-2 was drawn for the clear comparison between a traditional remote control and gesture control. Many studies (Billinghurst & Buxton, 2011) show the efforts in designing a user interface using gestures, which reflects that the use of gestures as a controlling method is beneficial.

Figure 2-2. Two different interaction model comparing the direction interaction between a user and product (right) and when it is restricted by a remote control (left)

CHAPTER 2.

20

Microsoft Kinect

Kinect is a motion sensing input device developed by Microsoft, and it is used for Xbox game consoles (Figure 2-3). Based around a webcam-style add-on peripheral, it enables users to control and interact with their console/computer without the need for a game controller, through a natural user interface using gestures and spoken commands. Kinect is especially good at body tracking, starting from recognizing different limbs over the body, and it even reads different facial expressions (Xbox.com, 2016). Its advanced ability made it possible to be used not only in the field of the game industry but also in various projects (Borenstein, 2012).

Leap Motion

Leap Motion controller is a small box (W 80 x h 12.7 x d 30.5 mm), which can be connected to a computer with a USB cable. Leap Motion enables controlling of a computer using two hands (Figure 2-4). The device recognizes hands movement, with a viewing range of roughly 60cm hemispherical area around it. It uses two cameras and three infrared LEDs. Various applications are available in the field of Computer controls, Creative tools, Education, Experimental, Games, Music & Entertainment, Productivities & Utilities, Science, and VR (“Leap Motion”, 2015).

After trying out some of the Computer controls, Creative tools, Games, Music & Entertainment apps, it has shown that Leap motion provided interesting experiences of controlling a computer without the presence of any tactile interface. However, understanding how the 3D hand is mapped into 2D monitor space was hard. The limitation in reading range did not provide much of the flexibility in performing gestures in 3D space. It felt rather passive when compared to the other devices.

Gesture control devicesSome renowned Gesture control devices such as Microsoft Kinect, Leap Motion, MYO band from ThalmicLab were researched and tried out in this research phase for the better understanding of Gesture control, including its benefits and pitfalls. Despite its prevalence, Nintendo Wii was excluded as it does not perfectly match to be an example for this project. The main reason was its association with a physical controller with buttons and interfaces. More Gesture control enabling devices were briefly looked into for the better understanding of the Gesture control devices and its market.

Figure 2-3. Microsoft Kinect: the device and its usage Figure 2-4. Leap Motion: the device and its usage

RESEARCH

21


MYO band

MYO band is an armband that allows Gesture control. Being worn around one arm under the elbow (Figure 2-5), it detects gestures such as double tapping, spreading fingers, making a fist, swiping to left and right as basics. The device recognizes such gestures by detecting muscle activities transmitted from the fingers to the arm (“Myo Gesture Control Armband”, 2015).

MYO band gave interesting experience from the first glance. It was fascinating that the device being worn around the arm, it recognized subtle hand and finger gestures that seemed irrelevant where it was worn. In order to perform well, it needed to be well-configured in the first time use. It was notable that it had an unlocking gesture which prevents unwanted events happening by detecting gestures accidentally. There were a few drawbacks of MYO device; the first was that it did not have a big language library, but only six types of gestures set by default. The gesture was not complicated, and it was performing tasks well as far as the device recognized the gesture, which was often not the case. Secondly, it could not avoid having a fatigue issue as it required putting the arm in the specific position to start.

Other devices

When more Gesture control devices were looked into, other various types of Gesture controllers were found as well (Figure 2-6). Arcus (“Arcus Motion”, 2016), Nod (“nod”, 2015), FIN (“Fin Robotics”, 2015), and Nimble (“Intugine Technologies”, 2016) are a ring-based Gesture controller that enables controlling of multiple devices with a finger movement. Gest (“Gest”, 2016) is a glove that works as an alternative to a mouse. Gesture remote (“Gesture Remote”, 2015) is a remote controller that works without a single button but only with gestures performed with it.

Figure 2-5. MYO band: the device and its usage Figure 2-6. Examples of various gesture control devices

CHAPTER 2.

22

Amazon Echo Amazon Echo is a home automation hub, introduced by Amazon. It sits on a table top, and makes various tasks and information accessible by voice, similar to Siri in iPhone.

It is compatible with a light system (e.g. Philips Hue), Smart plugs (e.g. Belkin Wemo), Sonos sound systems, Smart hub (e.g. Samsung SmartThings) and IFTTT service. It can access and provide information about personal schedule or weather, without a user having to type for certain keywords to a computer. It provides a new platform to achieve certain tasks in daily life, starting new types of interaction at home. This device is however considered to have more limitation than Siri.

Limited voice recognition range is one example. Different from Siri, Echo is a static object sitting on a table, while engaging with various tasks and appliances spread in a house (Figure 2-7). Having to approach to Amazon echo can create annoyance. Echo also has demonstrated hit-or-miss results when asked common questions that users would expect better answers to (Greenwald, 2015; Snell, 2015).

2.1.3 VOICE CONTROL

Voice control is also seen as Natural User Interface (Ballmer, 2015; Bolt, 1980), utilizing natural human communication skills. While both serves as a communication method, the fundamental differences would be that the one relies on verbal communication, while the other relies on non-verbal one. When it comes to human communication, verbal communication takes a major part of it, and non-verbal communication, such as gestures or facial expression could add more contexts to make the communication richer. It suggests the possibility of the Gesture control as a complementary method for the voice control. Providing a different medium that complements or substitutes an existing hardware interface, it was thought to have implications for Gesture control. Amazon Echo and Siri in iPhone were looked into. Mainly, how the users perceive Amazon Echo and SIRI were brought into focus.

Figure 2-7. Amazon Echo: the device and its usage

RESEARCH

23


Siri Siri is a voice recognition system implemented in iPhones launched after 2011.

It is very well-known and easily accessible interface. Siri utilizes human voice as an input, which makes it as a Natural User Interface. Siri case is considered highly relevant to this thesis project, because of the fact that it brought new controlling method that eliminates the need for any third-party controller to connect a user and a product into a daily living. According to one survey, as little as 15% iPhone users are using Siri, which concludes a very low rate of adoption for Siri (“Siri Archives - Intelligent Voice Blog - Speech Technology”, 2013).

The concept of Siri looked very promising at first, being able to control a phone hands-free definitely gives beneficial use cases. But the low adoption rate suggests that something hinders Siri from achieving acceptance. According to a research about the acceptance of Siri, it is said that the users of Siri go through four stages: Initial excitement, Optimistic curiosity, Frustrated exploration and Acceptance & habit formation. The research says that although users start to use Siri with interest, however Siri eventually fails to meet the expectations of the users, disillusions them and then losing its charm quickly. The reasons for this were: Talking to Siri did not feel like talking to a person, as it only understands short commands that were over-enunciated and spoken loud; Siri did not learn and remember from the past conversations; Siri did not support many attempted commands; the users did not know Siri’s capabilities; and Siri also misspells or misunderstands words (Auld, 2012).

To specify more in details, first of all, Siri is not smart enough. It can recognize what users are literally saying, but it lacks the ability of intelligently understanding the context and the user needs, which often ends up not understanding the given task. It requires the user to give a very specific task in a way that Siri understands (Figure 2-8), which then again dissatisfies the user and evokes annoyance (Honan, 2015). The essential idea of using the voice control, to make people more convenient is then hindered in this case. Second of all, using the voice control in public is also alleged of raising social awkwardness in public (Yarow, 2016). Lastly, the imperfection of the voice recognition creates some annoyance as well. The voice recognition technology is not perfect yet, thus it malfunctions from time to time. It could simply not understand what is said or perform a wrong task that was not asked for. Another technological constraint would be the sensitivity issue. Siri, or voice control is dependent on the quality of input. At a noisy environment, it is very likely that it will not function effectively.

Figure 2-8. Siri loaded on iPhone and some of its use cases

CHAPTER 2.

24

2.1.4 SMART LIVING

After the research focusing on Gesture control and voice control, the meaning of Smart living and related topics was looked into, to bring two themes together.

DefinitionThe meaning of Smart living can be summarized as advancements in people’s living brought by various technologies. When it comes to Smart living, terms such as Smart home, Smart house, or Smart environment are often associated when it comes to a daily living space. Smart home is defined as a residence equipped with information technology which anticipates and responds to the needs of the occupants, working to promote their comfort, convenience, security and entertainment through the management of technology within the home and connections to the world beyond (Aldrich, 2003). Often, Smart living is associated with Home automation. The concept of Home automation appeared since the 80s and includes well-known domains such as: HVAC (i.e. heating, ventilation, and air conditioning), Lighting, Shading, Security, Audio-visual, intercom, etc. (“Home automation”, n.d.).

A literature review regarding Smart living (Solaimani, Keijzer-Broers, & Bouwman, 2013) show more broad areas that are influenced by Smart living. The innovations apply to various areas, ranging from Home automation, Healthcare, Medical, Energy and sustainability, Education which are familiar to Telecommunication, e-Commerce, Gaming. According to the literature, Smart technologies cover topics such as: Network technology, Sensor technology, Artificial intelligence, and Communication & control. Home-remote control, wearable technology, motion sensor, pattern recognition etc. have been discussed as its subtopics, and this shows more relevance to Gesture control. Especially regarding studies in the field of Design and development, topic such as sense of control, interfaces such as motion tracking, gesture recognition and speech, interface personalization, detection/recognition of human intentions, feelings, situations and activities were the focus. These various focuses suggest the relevance of Smart living and Gesture control. And this seem to imply that the objective of this thesis, to successfully integrate Gesture control into Smart living environment, is a feasible one to achieve.

ExamplesFor a better understanding of Smart living and related technologies, examples of products that are associated with Smart living were collected (Figure 2-9). Network technology, IoT (Internet of Things) in other words, is notable especially when it comes to home automation. IoT technology connects an individual product to communicate to each other to yield an outcome desired by the user. Various sensors, actuators, and products that are connected and

communicating each other detects user needs at certain situation and time and automate various household tasks. Smart home hub products, such as Samsung SmartThings (“Smart Home. Intelligent Living.”, 2016) uses a hub that enables the whole system, various sensors that detect arrival, water leak, temperature, etc., and actuators or outlets that can execute various tasks based on the rules created by the user. Such hub products are often paired with applications, which allows users to have control over such systems regardless of the location. Similarly, smart plug products allow users to have full control over the devices connected to a smart plug regardless of their presence at home.

NEST (“Nest”, 2015) is a good example of home automation with more intelligence. Different from typical home automation products which users are always required to set rules, Nest learns from user patterns and automation rules are set based on the patterns eventually, with much less user effort.

In healthcare, much effort has been made for Ambient Assisted Living (“ACTIVE AND ASSISTED LIVING PROGRAMME”, 2016). Various fitness products to monitor and track health and activity status of the user is also already familiar.

User ReactionsDespite the aforementioned benefits of Smart living technology, it seems that it is not yet dominant in the market. When not considering the high usage of Smart devices such as Smartphone or tablet PCs, the use of Smart products in more holistic context seems to be limited, similar to the penetration of Gesture control.

People show contradicting opinions toward Smart living, also

Figure 2-9. Various domains and products that are perceived as relevant to Smart Living

RESEARCH

25


creating controversies. Some people are willing to accept, the others are rather skeptical about the new technology, and worries about it might have a bad influence on human values (Harper, 2006). Moreover, this was considered to have an influence in introducing Gesture control as well.

2.1.5 PRELIMINARY CONCLUSION

Insights and conclusions were drawn preliminarily, based on the desk research on the Gesture, Gesture control (devices), Voice control, and Smart living.

Gestures and gesture controlGestures make verbal communication more effective, while it can serve as an independent communication method as well. This implies that it can also serve as a communication in human computer interaction, and a number of researches and devices mentioned earlier supports this thinking. One of the benefits of gesturing to help the speaker to think better, suggests that adopting Gesture control could help the controller to think better. It will encourage more natural and intuitive interaction between human and product. In general, using gestures as an interface can provide emotional benefit, such as being fun, as seen in the example of Kinect. Leap Motion shows that limiting gesture space can hinder the experience. However, this aspect is premature to imply anything for the project as both Kinect and Leap Motion are mainly used as an entertainment device.

The research about the voice control leaves some implications.

Some pitfalls of the voice control, such as having to use very specific commands and being sensitivity to noise, suggests the possibility that Gesture control can be an alternative, or a supplement. The user experience for the Gesture control will be dependent on the gesture recognition, thus the technological aspect of it should not be neglected. The Siri case implies that social awkwardness should as well be considered when designing for Gesture control, as gestures are as noticeable as voice.

Smart living and Gesture controlIntroducing Gesture control in the context of Smart living seems to correspond in general.

Smart living covers a broad area which makes it hard to strictly define its domain and influence. One clear fact is that it makes life more convenient and comfortable, by automating tasks, reducing steps and effort, connecting physical products to talk to each other, enabling easier communication, making information accessible, and making controlling easier. As Smart living pursues various values and covers diverse domains, it seems to provide plenty of space for

Gesture control to be introduced.

The research on Smart living revealed some concerns as well. Some people show worries and anxiety toward Smart living, concerning laziness or dehumanized lives. The negative opinions should be taken under consideration for the application of Gesture control.

In order to have better understanding of Smart living, and see the potential of Gesture control to be implemented in the given context, it was concluded to interview potential users.

Their understanding and attitude toward Smart living, and the possibility of accepting Gesture control in daily life will be the main concerns of the interview.

CHAPTER 2.

26

OBJECTIVE

The user research aimed to discover opportunities for the Gesture control in the Smart living context by interviewing potential users. The research topics regarded: users’ living contexts, lifestyles, experience with Gesture control and Smart living, and user needs for the Gesture control, along with user profiles.

• User profile User profile information was collected to determine the target user group in the latter stage of this thesis project. The interviewees as ‘potential users’ were asked to provide information such as age, nationality, living arrangement, technological acceptance, home appliances, and Gesture control experience.

• Lifestyles This topic aimed to understand the current lifestyles of the potential users. It discussed co-residents, housing situation, tasks and activities taking place, and appliances being used and any accompanied interaction issues.

• Gesture control experience This part discussed the Gesture control related experience of the participants and their reflection.

• Smart living experience This topic aimed to see the current status of Smart living along with the expectation of the users.

• Beneficial scenarios using Gesture control After probing lifestyles, various physical or emotional benefits of using Gesture control at home were discussed additionally.

METHODS

ParticipantsIt was a qualitative study in which nine people took part. When selecting the interviewee, criteria such as age, gender, nationality, living arrangement, technology acceptance, the number of appliances or smart devices at home, previous knowledge or experience with Smart living, and Gesture control were considered and kept diverse. The participants’ technology acceptance was especially thought more important as it was thought to heavily influence the attitude toward Smart living realized by various technologies. The age varied from 20-40, and three nationalities were involved; Danish, Swedish and Chinese. All of the participants had a design or design research related background, either from their education or industry.

ProcedureInitially, the context mapping method was tried out as a pilot study, using the sensitizing material (See Appendix A. Sensitizing material). This process led to an insight that an interview at a real living space with appliances would be more valuable, as it was hard to picture the real-life interactions with descriptions on paper. Each of the interview sessions took approximately an hour, and took place at the participant’s residences. To thereby ease the answering of the participants and ensure for rich data, as participants would have the ability to point out or act out what they wanted to refer to.

The first focus was the lifestyle – activities and related housing appliances. The participants were asked to introduce his/her housing, living arrangement, or co-residents at the first part of the sessions. Moving to different spaces, the purpose and activities being one in the spaces were questioned. At the same time, the participant was asked to introduce appliances in the rooms, and how and by whom they are utilized during aforementioned activities, including any usability or user experience issues.

The second focus was Gesture control. The participant’s experience with Gesture control and the reflection was discussed. Afterwards, the participants were asked to share opinions about the benefits of Gesture control in their living context.

The third and last focus was Smart living. The participant’s experience with related products, reflections, and expectations were discussed. The full list of interview script can be found in Appendix B. User Interview Guide. Audio recordings were made through all the sessions. And pictures or videos were taken for any notable moments.

USER RESEARCH2.2.

RESEARCH

27


AnalysisThe interview results were summarized (Appendix C. User Interview Summarized Results), and valuable information and statements were extracted and used for the analysis.

Firstly, correlations in the user profile criteria were analyzed.

Opportunities for Gesture control were found from the results of the two research topics about users’ lifestyle and benefits of Gesture control. The lifestyle topic revealed various interaction issues with home appliances, which led to the insights for the opportunities for Gesture control. The Smart living experience and expectation of the participants led to the implication for the Gesture control to be integrated into the Smart living context.

RESULTS AND ANALYSIS

Typical usersThe participants’ profiles were as seen as in Figure 2-13. The majority of people were living in a nuclear family (i.e. an elementary family, a family group consisting of a pair of adults and their children), with intermediate number of household appliances, with intermediate Smart living experience, with few experiences using famous Gesture control devices such as Kinect, or Wii.

Smart LivingExperience with Smart livingThere were three expert, one intermediate, and four novice participants. The expert had an advanced home automation, smart home system, along with a few smart products and gadgets (e.g. Philips HUE, Sonos speaker, smart plug, fitness tracker, smart body scale, Apple TV, etc.) at their residence. The intermediate had a smart product such as smart TV, smart speakers, or a baby monitor. The novices did not own any smart products, except for their tablets, or smartphones. In general, Smart product ownership and technological acceptance were correlated. However, this was not the case for the young participants. Although their technological acceptance was high, due to the living arrangement and budget, they could own less smart products.

Expectation/Attitude Two important aspects of Smart living mentioned by the participants were automation and intelligence. While automation meant performing tasks based on the requests, intelligence meant performing tasks by predicting user needs. The levels of expectation were various depending on the participants.

The experts revealed positive impressions about their Smart living experience. They were impressed and satisfied with the current smart technology. They are open to explore new possibilities and looking forward to see more features to be realized in the future (except for Smart TV, the reasons given behind this were; Media

box providing more contents, own operating system with poor compatibility and connectivity with other devices, remote control being easy to be accessed and providing more functions over Gesture control). Their expectations are seen more realistic, and achievable in the close-future. They often had specific functions, or technologies in mind that can help their life tasks easier. Their better understanding the current state of Smart living technologies were thought as the reason behind this.

“When I walk out of my door I always user the voice command to turn off everything in the living room.”

“I just use my phone to control all the light before sleep, because Amy cannot hear me in the bedroom.”

“When it is quite, it works really good. I am really satisfied and impressed, and I use it everyday.”

“It is connected to the network, you can control over your phone even when you are away from home. I would like to connect more things like iron, for the safety.”

Figure 2-10. Quotes from the expert users

On the other hand, the other participants with little or no experience with Smart living tend to show a negative attitude toward current technologies. They were not willing to spend time, effort and money to buy and learn new Smart living technology unless they see big benefits in terms of necessity and practicality. They expected big changes, although they were not explained in details.

“If it could save money, make my life easier, and make the world better place, then it will be definitely interesting.”

“For me, gesture control has to do really smart for me. Like chopping up tomatoes, like a robot, really automated.”

Figure 2-11. Quotes from the novice users

Additionally, they raised concerns about laziness, dehumanization, and disconnection in family caused by the Smart living technologies, which is in line with the previously discussed literature findings. One participant said that he would accept new technology at work as far as it adds to efficiency and make tasks easier. But efficiency, ease of use, etc., are not the only thing that he was concerned when it comes to his living space.

“I would like my working life to be more convenient, but not my home.”

Figure 2-12. Quote from the novice user showing potential.

CHAPTER 2.

28

Figure 2-13. The overview of the user profile distribution

RESEARCH

29


Some people thought that it is untimely to accept Smart living. They saw current technologies were immature, and not advanced yet to bring the benefits that they expect. High price was another reason. Some technologies (e.g. controlling the light with a smartphone, controlling appliances with clapping) were considered as gimmicks, sometimes even without any experience. They also accused the technologies of making people lazy, and bring negative influence to family experience at home.

The difference in the expectation for Smart living of both types of the users can be seen (Figure 2-14).

Some quotes indicated the possibility that the conservative people can change their attitude depending on their needs (Figure 2-15). This leaves the potential for the introduction of gesture control.

Figure 2-15. Quotes Showing New Opportunities

“ I was skeptical at first. Then absolutely met my expectation.

I can recommend Sonos.”

- Technologically conservative user

“I try to keep my life minimal and learn less. But sometimes,

smart product surprises me. I learned to like something that I did not expect.”

- Technologically conservative user

““My study definitely influenced me, but I also see my non-techie friends

impressed with the voice control and they say they would like to have it. “

- Technology expert

“ My kitchen is the most non-techie place, and I would love to preheat the over with my phone, or with voice control, adjust the stove, etc. “

“ I would definitely accept those technologies. I am already one of a kind to accept those… I will do it as soon as possible. “

“ If it were automatic, it would be cool. It is hideous to come over just to control the heating system. “

“ Smart living is like your own butler. I will definitely use technologies that make my life more comfortable if it is affordable.”

“ It could do very simple stuff, but they are what I do the most, turning on the music and controlling the volumes.”

“ It is about life quality. Gadgets will get you more disconnected with my family.”

“ If I drop something, I will pick it up later. When your hands are full, you just use your elbow. But I don’t see it as a big problem.”

“ I am trying to keep my life simple, I don’t want to learn more to use it.”

“ I think it is still very immature. For me, gesture control has to do really smart for me. Like chopping up tomatoes, like a robot, really automated.“

“ I don’t think removing a bit of annoyance can make a big difference.”

Figure 2-14. Difference in expectations of the expert and novice users

EXPERTS NOVICES

CHAPTER 2.

30

Gesture Control

ExperienceThe Gesture control experiences of the participants were only limited to the entertainment industry. Four participants had an experience of using Gesture control devices such as Kinect (Xbox), MYO band, or Wii. Kinect was the most popular one. Three participants who have used the device thought it was fun to play with the device. They also thought Kinect shapes the game experience social and active, unlike other conventional games. Kinect was also evaluated to be more accurate when compared with Wii or Leap Motion.

AttitudesFor those who are more open to Smart living and technology, Gesture Control as a new control method was already interesting enough to give it a try, and either small or big convenience that gesture control can bring was worth for them. They were willing to accept more technologies to shape their daily lives bit more convenient and comfortable.

Technologically more conservative people are less likely to buy it unless it brings obvious benefits. They also had worries that it could be just a gimmick, without bringing any real, and valuable benefits.

Opportunities: Beneficial scenarios for Gesture controlVarious interaction problems were mentioned regarding home appliances. The problems discussed usability, user experience, technological constraints and wishes. Based on the valuable statements, situations that could benefit from Gesture control were discussed. Nine cases that show high potential for Gesture control to solve were identified (Figure 2-16), which could be grouped into three topics.

• Utilizing unavailable hands The first case is to overcome a short-handed situation, when a person is out of hands and not able to handle extra devices to control something. The second case is to overcome a dirty/wet-hand situation. When one has dirty or wet hands and he/she does not prefer to touch any device not to transfer any residue. This also includes the case of a touch screen based interface, that does not work properly when controlled with a wet hand.

• Distanced control The third case is to overcome a long distance, and the original controlling method is disabled. It includes a case when the user wants to control a product either that is not in the sight, or even very far away. The Gesture control can provide a possibility of controlling such a product while imagining target visionary. The fourth case is to overcome a rather short distance when the original control method is interfered, such as falling asleep, enjoying a movie, or not being able to reach the interface. The fifth case is to control over a distance due to any hazardous health issue.

• Better usability The sixth is to solve the usability issues of hardware interfaces. The Gesture control made simple, and easy can provide an alternative to a poorly made remote control with too many buttons. The seventh case is labeled as the Physical existence. This case is to remove the needs of finding a physical remote controlled, which can be annoying from time to time. The eighth case is to overcome the technological constraints of the old remote controls, which only works when aimed correctly to a target. The ninth case is to serve as a universal control.

Figure 2-16. Situations that can benefit from Gesture control

SHORT-HANDED

PHYSICAL EXISTENCE

LONG DISTANCE (DISABLED)

UNIVERSAL CONTROL

DIRTY/WET-HANDED

TECHNOLOGICAL CONSTRAINTS

SHORT DISTANCE (INTERFERENCE) USABILITY

HEALTH HAZARD

RESEARCH

31


Potential for Gesture controlIt was confirmed from the interviews that Gesture control could bring functional benefits; such as overcoming short-handed, or dirty/wet-handed situations, short or long distances, usability issues caused by traditional remote controls, a chance of misplacing any physical control, technical issues of connectivity of Infrared control, lack of universal control, and health hazardous situation.

Despite those potential benefits, however, it is concluded that some users might not buy the Gesture control, as very functional and practical benefits solely are not persuasive enough to them.

The meaning of smart livingWhile Smart living provides benefits in various domains, user reactions are diverse. It is common that users with more Smart living experience are more satisfied and open to accept new technology. The others thought the technology is pricey and still somewhat premature. It is interesting that both groups thought the current Smart living provide automation, and intelligence seems to be the future of Smart living.

Gesture control and Smart livingControlling with gestures are considered interesting and convenient for some people. However, it is concluded that controlling with gesture itself would not be considered smart. A complex Gesture control can offer more vocabularies mapped with functions, but it is doubtful that if such a system can be considered ‘smart’. In order to be a smarter system, Gesture control should aim to be intuitive, easy to use, and natural.

The expectation on intelligence in smart living is something to consider.

This leads to a conclusion that more unconsciously executed gestures have more potential to be implemented for the sake of ‘smartness’. It also meets the criteria to be intuitive, easy, and natural. When the typology of gestures (Figure 2-17) is looked into, the more left the gesture is, the less mental effort is required. In that sense, action/behavior on the left end, and it can indicate a task that the user is trying to perform. Based on this, it is thought to have a potential to be translated as an input, and it is thought to be the most direct interface that requires the least amount of the mental effort. However, more difficult gestures can convey more direct and sophisticated meaning. Thus, both contradicting properties would have to be balanced well.

CONCLUSION2.3.

Figure 2-17. The relationship between gesture types, required mental efforts and the potential to be used in an intelligent system

IDEA GENERATIONInitial ideas are presented in Chapter 3.

The first ideation is based on the conclusion of the previous phase: The future of Smart living is intelligence, and Gesture control should be intelligent to be integrated. The idea to design a home automation system triggered by natural actions/gestures, turns out to be infeasible conceptually and technologically. The iterative process of the first ideation is briefly covered.

The second ideation starts by brainstorming the benefits of gestures and its application in a right context. This practice results in eight primitive concepts. After the evaluation of the concepts, “smartphone gesture control” is selected and continues to be developed.

3

CHAPTER 3.

34

3.1.

This initial ideation aimed to combine Smart living and Gesture control. It was based on the direct translation of the conclusion of the previous phase, that Gesture control should be intelligent to be integrated into Smart living. It focused on the realization of intelligent home automation, using actions and gestures. Moreover, the automation should be applied to the household tasks that does not add to or hinder the family experience.

This direction was pursued through a few iterations using various methods such as personas, role-playing, Wizard-of-Oz user tests, video visualization and more. However, this direction was withdrawn in the end as it was evaluated to be technologically infeasible or valueless to be developed as a design concept. The iterative process is summarized in this chapter. The process is covered extensively in Appendix D. Iteration 1: Iterative Process.

3.1.1 IDEA DESCRIPTION

This direction aimed to achieve smart/intelligent home through and the use of natural actions as gesture input. In essence, the concept is a home automation system that recognizes natural

actions and gestures and translates them into commands to execute automated tasks. Those intuitive gestural inputs include waking up from bed, entering a particular space, shivering or fanning to adapt to temperature. The utilization of actions, behaviors, and rather unconscious gestures minimizes the mental effort to remember the commands and make the system to be very intuitive. The system also understands the context: who the user is, where the user is, when it is, and what is being controlled. The system also seeks to be intelligent, by learning user patterns over time (Figure 3-1).

This idea intended to meet the needs of a specific type of the users named as Value-seeking hesitators. The value-seeking hesitator refers to people who pursue convenience and comfort but also traditional values such as family experience, thus careful about adopting invasive technologies. The term, value-seeking hesitator, was coined based on the analysis of the previous user research. And more information about this can be found in Appendix E. User Typology.

3.1.2 SUMMARY OF ITERATION PROCESS

Two user research sessions were executed to evaluate the preliminary idea. During the first user interview with the three

IDEATION 1: GESTURE TRIGGERED HOME AUTOMATION

IDEA GENERATION

Figure 3-1. Gesture-triggered Home Automation Idea

Mark, gets the smart gesture control kit.

Master control gesture allows the manual control.The gesture can be personalized.

The system detects the user waking up in the morning. It opens the blinds, turns off the alarm, and brews a coffee.

Mark installs the system using the desktop application.He selects products that he would like to use with the system.

The system learns the user pattern, registers it as a daily routine,then automates various home tasks.

If “Shivering” gesture is detected, the system raises the temperature of the thermostat.

The system asks to place the sensor unit (camera).

When the initial registration is done, the user can exploremaster control right away.

When the last person leaves home,the system will secure the house.

Mark, gets the smart gesture control kit.

Master control gesture allows the manual control.The gesture can be personalized.

The system detects the user waking up in the morning. It opens the blinds, turns off the alarm, and brews a coffee.

Mark installs the system using the desktop application.He selects products that he would like to use with the system.

The system learns the user pattern, registers it as a daily routine,then automates various home tasks.

If “Shivering” gesture is detected, the system raises the temperature of the thermostat.

The system asks to place the sensor unit (camera).

When the initial registration is done, the user can exploremaster control right away.

When the last person leaves home,the system will secure the house.

35


potential users, the design idea was presented with a video prototype to the three participants from the first user research. The participants gave positive reflections in general. Especially regarding the basic idea of the concept, participants appreciated the automation triggered by natural, contextual actions and the possibility of pattern-learning. The participants considered such a feature to correspond to intelligence, as the future of Smart living.

The second user research aimed to assess features and gestures for the concept, using interactive prototypes and Wizard-of-Oz method. This was done with two experienced designers. The features included few automation scenarios triggered by gestures and master control that enables the users to stop automated tasks manually. Moving hands up/down to control roller blinds were evaluated positively. Crossing arms and fanning to adjust thermostat were evaluated somewhat positive. However, the actions were not thought universal. Covering/cupping ears for speakers, putting a hand over eyebrows to dim the light were evaluated to be unnatural, and socially awkward. The master control feature (to point and make a personalized signature movement) was considered as a necessary function.

Camera based gesture recognition was seen to be the only solution to realize the concept, as people did not prefer any wearable gadget all the time while being at home. The use of such methods for few automation scenarios was thought not economical.

3.1.3 SUMMARY OF THE RESULTS

The concept was not considered feasible in the end, considering conceptual, technical and economical aspects.

Firstly, it lacked applicable automation scenarios. Complex schedules and many possible deviations in people’s daily living hindered to come up with many automation scenarios. Secondly,

users preferred camera-based gesture recognition over wearable devices, but field of view and the location of the camera were hard to determine. Thirdly, such technology was thought too expensive to realized only few possible scenarios. Furthermore, the intelligence of Smart living and the Gesture control conflicts in its essential characteristic. While intelligence is about predicting user needs and automating tasks, Gesture control requires a user to make an input.

Although this direction was not pursued in the end, it enriched the understanding of the application of Gesture control in Smart living context and led to the following design implications.

Using original actions as triggers are considered smart. A simple one-to-one mapping of gestures and function would not contribute much to Smart living. An idea to translate natural actions as a trigger for a task is considered smart. It also leaves a potential to be a pattern-learning system, which eventually leads to intelligence. This links with the future of Smart living.

Activation gesture is crucial to make automation happen.Participants were concerned about having one unified system that only uses gestures, as it could go wrong and mess up with life very bad, when the system is underdeveloped and gestures are misinterpreted. Therefore, the use of an ‘Activation’ gesture is considered necessary. One participant mentioned the possibility of using voice, instead of using a deictic (pointing) gesture to select an object to control.

Social awkwardness should be considered.Some gestures like shivering, fanning, cupping an ear were considered intuitive and natural. Some others such as covering ears, waving, and covering eyes were instead considered unnatural. Especially the ‘covering ears’ gesture was evaluated to evoke an awkward experience since it could be translated into ‘I do not want to listen to you’, if the gesture was made during a conversation with someone else. Social awkwardness should be considered in designing for a Gesture control, since the gesture is visible to others, and it could create bad experience if not carefully designed.

Figure 3-2. Iterations: Survey, user test setup planning, Wizard-of-Oz setup, and user test

Section 1 of 9

After section 1

Section 2 of 9

Gesture-controlled Smart LivingThank you for participating in this research. The objective of this survey is to gain insights in the field of Smart Living, regarding a new Smart Living system that utilizes a new way of interaction: Gestures. As a potential user, your input will be very valuable. Please spare a bit of your time and tell us what you think.

Gesture-controlled Smart Living system Storyboard

The video below describes a new system that enables Smart Living with a new interaction vision. Please watch the video, and tell us about your opinion.

Untitled Video

Continue to next section

CHAPTER 3.

36

3.2.IDEATION 2: BENEFITS OF USING GESTURES

This ideation focused more on the application of gestures in a right context. It was done because the direct translation of the insight from the previous research phase did not lead to a possible design concept. A property of gestures and benefits of using gestures were brainstormed to provide a base ground to initiate idea generation. This design activity was done using the WWWWWH method combined with mind mapping.

3.2.1 BRAINSTORM ABOUT GESTURES

The properties of gestures were brainstormed using the WWWWWH method. The method helped to analyze the problem by understanding the problem thoroughly concerning various perspectives (Boeijen, Daalhuizen, Zijlstra & Schoor, 2013).

3.2.2 BRAINSTORM ON THE BENEFITS OF GESTURE

The use of mind maps encouraged to analyze the problem and to come up with possible design solutions openly (Boeijen, Daalhuizen, Zijlstra & Schoor, 2013).

The two brainstorm sessions about gestures and its benefits led to the following insights. Gestures can show affordances.

Gesturing can show affordance as certain gestures are a part of actions related to a task that was to be accomplished (e.g. Rolling up/down blinds, approaching, picking up, grabbing certain things).

Gestures can communicate some visual traits more effective.Gestures seemed to be beneficial due to its visibility. Gestures enable the communication of certain types of visual qualities easier and efficient, compared to when only verbal communication is used.

IDEA GENERATION

Figure 3-3. Brainstorm about gestures using WWWWWH method and mind mapping

37


Those visual qualities included size and shape, but not colors when it comes to appearance. Furthermore, gesturing can contribute communicating tactility, haptic qualities, motion and speed very easily.

Gestures can help communication in a loud place or over some distance. The visual quality of gesturing can also be beneficial in loud places or situations when communicating only verbally becomes hard but in a visible situation. The examples would be: ordering at a noisy restaurant, at a party, when loud music is played, when cooking, and when communicating over a glass. Gestures can help to overcome noise at a short distance.

Gestures can serve as an easier communication method.Gesturing in general functions as a communication method either along with speech or as it is. The examples can be: sign language, babies using gestures etc. Also in everyday situations, gestures

were found to help people to think better, which is in line with the conclusions drawn form the desk research.

Gestures can be used very intuitively, with less mental effort.When it comes to being part of communication, gesturing is often done very natural, subconsciously or even unconsciously compared to the use of a verbal language.

By combining these benefits of the gestures with the findings from the first ideation phase, eight primitive design concepts were generated. Using the Strategy Wheel method, these concepts were then evaluated and compared with each other. The assessment was based on the criteria concerning values in the field of Smart living, Gesture control, and design concept. Through this process, the final concept direction was determined to be: “a smartphone Gesture control”.

Figure 3-4. Mindmap about the benefits of gestures

CHAPTER 3.

38

Mood translator for un/subconscious gestureThis mood translator detects the user’s emotions and creates a dialogue to reflect a day. It is realized by detecting any unconscious, or subconscious gestures showing emotions, such as frowning, yawning, cupping a chin in hand, or smiling. Being used by people who work a lot sitting in front of a computer, it is expected to help them to reflect and take care of their feelings during the day, which can normally be easy to be forgotten and neglected.

Interactive stress ball with context awarenessThis interactive stress ball made with flexible silicon material provides different sensory feedback, depending on the gestural input given. For example: if the user stretches or sheer the ball, it will play a sound of ripping paper; if the user throws the ball, it will play a sound of cracking glass, metal, or any other expensive things such as a laptop.

Fingertip writerFingertip writer enables making a note by writing with a fingertip on a palm and saves it on a digital device, such as a smartphone. This concept can help people to write texts and make drawings in a more intuitive way, by allowing them to do so with one’s fingertip instead of using a mediator such as a keyboard.

Gestural input for smartphoneThis is a gesture interaction concept when interacting with a smartphone. Some tasks when using a smartphone often involve specific actions/behaviors (e.g. making or answering a call accompanies picking up a phone and bring close to an ear, taking a picture makes people hold the phone in a specific stable way, and listening to music often accompanies plugging an earphone in). Such behaviors can be used to trigger the related function.

IDEA GENERATION3.3.

IDEA GENERATION

Figure 3-5.Idea sketch: Mood translator for un/subconscious gesture Figure 3-6.Idea sketch: Interactive stress ball with context awareness

Figure 3-9.Idea sketch: Fingertip writer Figure 3-10.Idea sketch: Gestural input for a smartphone

39


In this section the eight primitive concepts that were generated are summarized. In overview, the concepts were: Mood translator from unconscious/subconscious gestures; Interactive stress ball with context awareness; Fingertip writer; Gestural input for a smartphone; Context-aware cup; Kitchen utensil with gestural input; Baby gesture translator; and Visualization tool for the communication of visual qualities. Larger images are available in Appendix F. Idea Sketches.

Context-aware cupThis concept is a cup which can warm up its contents when held and wrapped around with two hands, and cool down when the upper rim is held because the content is too hot.

Kitchen utensil with gestural inputThis is a concept for a product similar to a food processor, controlled by gestures. Instead of a traditional button interface, the product detects gestures such as slicing, grating, mashing, and performs the according function. When it comes to slicing, the user can select the size/length of the outcome by varying the gap between each chop gesture.

Baby gesture translatorThis concept was inspired by babies using gestures even before starting to talk. (Pictures). They not only point, but also use different gestures to indicate a specific product. Baby gesture translator can help the parents to understand their babies better.

Visualization tool for communicating visual qualitiesThis is a transparent screen that helps communication between people especially about visual aspects such as shape, or size. The screen detects a person describing a shape or specifying a size by gesturing, and it will draw the according shape and size on itself, making it more concrete to be visual and to be seen.

Figure 3-12.Idea sketch: Tool for the communication of visual qualities.Figure 3-11.Idea sketch: Baby gesture translator

Figure 3-7.Idea sketch: Context-aware cup Figure 3-8.Idea sketch: Kitchen utensil with gestural input

CHAPTER 3.

40

These eight concepts were compared using the Strategy wheel (Figure 3-12).

The use of such a method contributes to a visual overview of different design solutions and enables comparing them in a quick manner (Boeijen, Daalhuizen, Zijlstra & Schoor, 2013).

Twelve evaluation criteria were used that were each represented by an axis in the strategy wheel, on a scale from 1 (very low)-5 (very high). The criteria were established considering the three aspects of: values as a design concept, values as a Gesture control, and contribution to Smart living.

EVALUATION OF THE CONCEPTS

3.4.

Figure 3-12. Strategy Wheel criteria for Concept evaluation

S T R A T E G Y W H E E L

IDEA GENERATION

41


C R I T E R I A

Value as a Design ConceptThis aspect evaluated each of the concepts as a design concept. Novelty/innovation, feasibility, and expected user acceptance were the criteria.

NoveltyEach of the concepts was evaluated concerning its novelty, in terms of how innovative each of one was.

FeasibilityTechnological feasibility of the concept was evaluated concerning current technology available in the market and the time expected to develop each of the concepts.

AcceptanceThis criterion assessed if a concept will be accepted and used by people eventually. Most importantly, it was considered that if the benefits of a concept predominate over the efforts required learning and using the concept. It was also considered that a concept is not bringing a sudden, radical change that is difficult to be accepted in the current market.

ImpactThis criterion concerned the impact each of the concepts for making changes in people’s lifestyle. Concepts that are more related to the necessity by solving existing problems, meeting user needs are given with higher score.

Value as a gesture controlThis aspect considered if the use of Gesture control is applied in a right way, so it is usable and does not create any stigma or negative emotions.

IntuitivenessIntuitiveness concerned not only the general intuitiveness, but also concerning different age groups.

MaintenanceThis criterion concerned if the user needs to be engaged in its maintenance and by how much.

Natural InteractionThis criterion concerned if the interaction is made natural, as opposed artificial and defined.

Easy to useThis criterion was to measure the amount of tasks that users have to deal with along with the complexity of them.

Social awkwardnessThis criterion concerned any potential social awkwardness that can be raised. Often, gesturing without given a proper context can be seen awkward in public, especially when Gesture control is not widely adopted yet. Being used in the right context, being subtle were considered to beneficial to solve the issue.

Contribution to Smart LivingSome criteria concerned the contribution of a concept in a Smart living context.

SmartAs the outcome of this project has to contribute people to live smarter, ‘smart’ was also included as one of the evaluation criteria. It concerned the general impression of the concept, which was about if it feels smart, rather than assessing based on any concrete idea of being smart.

Automation /IntelligenceWhile ‘smart’ being a very general criterion, which can regard many different aspects, this criterion specifically concerned automation and intelligence perspective of Smart living. It considered to which extend a concept contributes to this field.

Genuine valuesThis criterion concerned whether a concept genuinely can add or create values in life over time, instead of being a gimmick with ephemeral benefits.

CHAPTER 3.

42

Mood translator for un/subconscious gestureThis concept detects unconscious, or subconscious gestures that can also be subtle, and because of that, it does not require the user to put much effort to make the system work, or interact with the system.

Thus, the concept earned high scores in intuitiveness (4), maintenance (4), easy to use (5), natural interaction (4), social awkwardness (4), smart (4) and automation/intelligence (4).

However, it scored low in novelty (2) and feasibility (2), mainly due to: the possibility of limited gesture library to detect, people’s different tendency to express emotions, and subtlety of the gestures to be detected. Genuine value (3) and impact (3) were evaluated fair, but not superior, since it does not give functionally practical benefits. Although taking care of the personal emotions can also be a good perspective in Smart living.

Interactive stress ball with context awarenessThis interactive stress ball concept does not require a complex gesture, nor gesture recognition, thus the feasibility, maintenance, easy to use scored very high (5).

The acceptance (4) scored high as well, as it should be a small and fun object that is not expensive to buy. A stress ball is already a common object to use.

However, it can make other people recognize that the user is in stress, so the social awkwardness and natural interaction criteria were scored fair (3). It was considered fairly new and fun, so the novelty scored high, but it feels like a gadget or a toy that is not very necessary to life, thus the impact, and genuine value scored very low (1). It was not considered Smart nor intelligent, as it works based on the direct one-to-one translation of a gesture input.

RESULTS

IDEA GENERATION

Figure 3-13. Idea sketch: Mood translator for un/subconscious gesture Figure 3-14. Idea sketch: Interactive stress ball with context awareness

For each of the eight concepts the results of the strategy wheel evaluation are summarized here.

43


Context-aware cupThis was seen as a rather novel concept, with high feasibility due to its simple gesture library and required technology.

The criteria considering gestural interaction scored fairly good, as it uses a very natural interaction that is commonly used by people.

However, it was not evaluated to be smart, intelligent, or influential. It can have a too limited scenario as well. The genuine value was evaluated very low accordingly.

Kitchen utensil with gestural inputThis concept utilizes one of the benefits of gesturing, to describe visual property such as size or form easier than using language. Moreover, a kitchen is an area where hands get wet or dirty, which makes the gestures highly potential to be implemented.

However, the smart, automation/intelligence criteria did not get high scores, as gestures work as an alternative method, without adding any big advancement but only a bit of convenience.

The user acceptance was questionable, since implementing gesture recognition function in such a kitchen utensil can cause high price, and the added value for this concept might not be enough to compensate. As this concept has an influence in a very limited space and limited situation, its impact and genuine value were also evaluated low.

Figure 3-15. Idea sketch: Context-aware cup Figure 3-16. Idea sketch: Kitchen utensil with gestural input

CHAPTER 3.

44

Fingertip writerFingertip writer utilizes very intuitive gesture of writing, without any other thing but a finger and a palm. It can be beneficial and fun when traveling. Thus, the novelty, acceptance, impact, intuitiveness, easy to use, and natural interaction scored very high (5) to high (4). Its feasibility and maintenance criteria scored fair (3). The technology to do this without any bulky device on a palm or a finger is not very clear. It does not have any automation feature, so the criteria automation/intelligence, smart and genuine value scored fair (3) and low (1). Writing on a palm with a finger is visible to others, although the interaction itself is not awkward. Thus, the social awkwardness was evaluated fair (3).

Gestural input for smartphoneThis concept had a fairly higher score in all of the concept, gesture, and Smart living aspect.

The input gestures are similar or identical to the original actions. Such actions represent tasks, providing a clue of what is being done. Thus the actions can be recognized and translated into commands. The gestures derived from the original actions offers another benefit; it can reduce a step in smartphone interaction, by executing a function without having to interact with a touch screen interface. In this sense, easy to use, natural interaction, and social awkwardness were evaluated fairly high. It is also considered relatively smart, and it also brings automation to some extent, although not intelligence.

Some subtle gestures can be hard to be recognized while some gestures are technically feasible and already introduced to the market. Thus, the feasibility was evaluated as fair (3). It shows benefits by saving efforts to interact with a smartphone. However, if it will bring a genuine value to daily life was questionable, due to its limited application area, which is a smartphone.

Figure 3-17. Idea sketch: Fingertip writer Figure 3-18. Idea sketch: Gestural input for a smartphone

45


Baby gesture translatorThis concept was evaluated good in terms of the utilization of gestures. It uses the benefit of gestures as a non-verbal communication method. The gestures are what is already being used, in a very natural context and situation.

It is not just intuitive for the babies, but also, make the parents’ lives easier. The user acceptance was evaluated high since parents are often willing to invest in baby products.

However, the feasibility is questionable. Baby gestures might not be universal, depending on culture, or the environment. There might be so many vocabularies to put in the system. Furthermore, as the parents often can understand what their baby is trying to communicate, and if the parent cannot understand, the system as being a computer might not be able to understand either. Learning to understand and communicate with a baby is also a joy of parenting. In that sense, the genuine value of the concept was not evaluated great, despite the functional benefits.

Visualization tool for communicating visual qualitiesThis concept utilizes the benefit of gestures properly, to communicate certain visual qualities (shape, size but not color) easier. It also utilizes natural interaction. in that sense, it can be considered that it is a new type of smart product that enables some drawing and communication tasks easier. Thus, novelty, intuitiveness, natural interaction, and smart criteria had good scores.

Having a screen to control made the interaction very natural, so it would not create much social awkwardness at all.

On the other hand, feasibility was evaluated low, as it involves many technologies that are not simple to build, such as: gesture recognition, projection to a screen and screen using gesture as an input.

Similarly, maintenance and easy to use was scored fairly low. Also, this concept lacks automation feature thus it was not evaluated to be intelligent.

Figure 3-19. Idea sketch: Tool for the communication of visual qualities. Figure 3-20. Idea sketch: Baby gesture translator

CHAPTER 3.

46

Final DirectionThe gestural input for a smartphone was concluded to be the final concept direction.

The other finalists included the kitchen utensil with gestural input and baby gesture translator idea. The kitchen utensil idea was discarded since it involved very limited types of tasks taking place in a very specific space at home. The baby translator idea was considered as less relevant to Smart living. It definitely can help living by helping interaction between baby and parents, however, it was not considered intelligent nor contributing to lifestyle.

A smartphone that uses gesture input to trigger certain function was evaluated to have various benefits regarding intuitiveness, ease of use, convenience, and natural interaction.

The concept was considered convincing that it could create

more concrete scenarios of solving existing problems, such as: annoyance dealing with an overwhelming amount of applications of a smartphone; and annoyance of putting an effort to lose real experience while looking for and launch a right camera application to capture a precious moment in life. It was also considered to have a potential of bringing back old haptic interaction with old Feature phones (as opposed to a smartphone and modern feature phones with a touch screen, a mobile phone with physical buttons as an interface which incorporates features such as the ability to access the Internet and store and play music but lacks the advanced functionality of a smartphone (“Feature phone”, n.d.)), making commands while not looking. Considering the high prevalence of a smartphone along with the frequency and the amount of time spent on the use of Smartphone, it was seen to have a potential to be connected with a Smart living. It was also seen to have a potential of being an intelligent system which could learn patterns from a user similar to Siri suggestions (Clover, 2015).

PRELIMINARY CONCEPT: SMARTPHONE GESTURE CONTROL

3.5.

Figure 3-21. The final concept direction sketch

IDEA GENERATION

47


Novelty of the conceptRelevant works were researched. This aimed to get basic understanding regarding the novelty of the concept.

Samsung’s gesture trigger functionsSamsung Galaxy S4 offers a functionality called Air view, Air Gesture. The Airview features allow the users to preview some information by hovering fingers over the screen. The Air Gesture feature allows the users to activate certain functions by making specific actions or movements without touching the screen. It includes functions such as Quick Glance, Air Jump, Air browse, Air move and Air call-accept (“How does “Air View” feature work?”, 2015). Each of the functions offers scrolling through a page, moving between pictures/pages/music tracks, moving app shortcuts or calendar events, answering incoming calls respectively (Meese, 2014). This feature is expected to help users in a situation such as having wet or messy hands during cooking, or wearing gloves (“Samsung GALAXY S4 - Life Task”, 2015). The functions are triggered by detecting hand movements in a relation to the phone. It is known to be realized by two types of sensors, proximity sensor and capacitive sensor (“Proximity sensor on Galaxy S4 (Air Gestures)”, 2015).

iOS Switch ControlIntroduced in iOS7, Apple’s “Switch Control” is an accessibility feature that allows users with very limited mobility to control iOS touch screen devices with a click of a button, movements of the body, or any number of alternative input or other adaptive devices (“iOS: Switch Control helps you navigate your iOS device - Apple Support”, 2015; “How A Simple Apple Feature Called Switch Control Is Changing Lives”, 2016).

With Switch Control users who are unable to use the touch screen can navigate the device, select any screen item, and even emulate more complex gestures (e.g. dragging, pinching) by simply clicking a button, blowing, blinking or using other alternative inputs (Tecla, 2016).

The two aforementioned related works provide valuable examples on how gesturing can help the interaction with a smartphone, and overcome some troublesome situations. Some features are very similar to the initial idea. However, the essential design approaches differ. Samsung Air Gesture maps a function with a gesture, and the association is very easy. It serves as an alternative control method to replace the original interaction that temporarily cannot be done due to any constraint (e.g. wet hands).

On the contrary, this design concept started off from the thinking to make use of the natural behavior as a source of inspiration. A part

of actions that are already performed in life is implemented as a gesture input to trigger a function. It is not a mere substitution for another control, but it aims to work as a short cut, removing steps to perform a certain task. It also reduces mental effort to think about the mapping. Furthermore, the concepts provide more accurate and smart features, by using more that just gesturing in the ‘air’. The orientation of the phone, location of the fingers, comparison between the previous and current status of the phone along with the proximity information will make the interpretation of the user needs more intelligently.

Figure 3-22. Samsung Air Gesture (“BoeBoer - Phone & Tablet’s Manuals Review & Download”, 2015)

Figure 3-23. iOS Switch Control

CONCEPT DEVELOPMENTChapter 4 covers various research and design activities to develop the initial design concept. The process goes through three iteration cycles. The first iteration polishes a gesture library through user tests. The second iteration introduces the design of wireframe concepts and its evaluation with the users. The third iteration covers the development of one specific function of all; Remote control using a smartphone in Home environments.

The design implications lead to the final design, which is covered in the next chapter.

4

CHAPTER 4.

50

4.1.ITERATION 1: SMARTPHONE FUNCTION-GESTURE MAPPING

During the first iteration, a few selected smartphone functions were mapped with gestures.

First of all, the tasks that we do with a smartphone were brainstormed and categorized. It led to a list of basic smartphone functions. The functions were classified based on the presence of gesture affordance. Thereafter, relevant gestures were brainstormed through Interaction Prototyping, and it led to a preliminary selection of gestures. This preliminary gesture set was evaluated with a user test.

4.1.1 SMARTPHONE FUNCTIONS

Smartphone functions were brainstormed and categorized. By expanding the knowledge on the various functions of smartphones, it aimed to identify potential functions for the Gesture control.

The short brainstorm was done regarding smartphone tasks, in association with places, activities, time, tasks, objectives of the use (Figure 4-1). It was concluded that an exploration of smartphone applications was necessary, as the applications execute and realize many functions of a smartphone. Applications highlighted as essential in Apple app store (Apple, 2016), and apps nominated as the most used apps from a web search (Tweedie, 2016; Frommer, 2015; Sacco, 2015) were clustered based on its service categories (Figure 4-2).

Whereas there are numerous applications available, the essential tasks that we perform are limited. The essential tasks included: Writing, searching, calling (talking), checking date/time/calendar, uploading a picture/file, mobile banking, games, recording a voice/sound, taking a picture/video, browsing (web), browsing photo album, streaming a video, playing music, playing games, remote control, map/orientation, alarm and typing a password. These functions were thus shortly explored to connect with relevant gestures, as described in the next section.

4.1.2 FUNCTIONS WITH GESTURE AFFORDANCES

A short brainstorm revealed that only some of the smartphone tasks can be associated with gestures, and those tasks were seen to have gestural affordances.

The current Gesture control concept cannot substitute all of the touch screen control, because of its diversity and complexity. For example, people would bring a smartphone close to an ear when starting a call, but it is hard to imagine what types of gestures are performed when they are playing games, uploading a picture, or using a timer. The results of this short brainstorm are organized in Table 4-1.

Thereafter, gestures were brainstormed for the twelve functions with gestural affordances. Interaction Prototyping (Boeijen, Daalhuizen, Zijlstra & Schoor, 2010) was done by acting out gestures with a real smartphone to envision the Gesture control (Figure 4-3).

CONCEPT DEVELOPMENT

Figure 4-1. Application categorization process

Table 4-1. Smartphone tasks based on the presence of gestural affordances

Figure 4-3. Interaction prototyping using a smartphone

51


Figure 4-2. Application categorization process

REFINED LIST OF FUNCTIONS

Most of the gestures were the direct translation from original actions performed to accomplish an intended task (e.g. Lifting a phone with hand to start using the phone, bring the phone to an ear to start calling, plugging earphones to listen to music). Snoozing or turning off an incoming call or an alarm were some of the exceptional cases. Still, they were still considered relevant to the people’s life experience. People would often flip a phone when they do not want to be disturbed by the screen after snoozing a call, and it would work similarly with alarm as well. Orienting a map gesture came from an experience that iPhone would often ask its user to wave in 8 figure gestures when GPS interference occurs. The full brainstorming contents can be found in Appendix G. Interaction prototyping. The brainstorming session led to a preliminary set of gestures regarding smartphone tasks with gestural affordance. The results are introduced in the next section.

CHAPTER 4.

52

4.1.3 PRELIMINARILY SELECTED GESTURES

One or two gestures were preliminarily selected for each of the twelve functions: Make a call,Send a message, Take a photo/video (with one hand/two hands), Record voice/sound, Play music, Stream a video (with one hand/two hands), Enter the passcode, Remote control, Calibrating a map, and Snooze the alarm.

MAKE A CALLTo initiate a call

Brining the phone to an ear

STREAM A VIDEO (ONE HAND)To stream a video lastly seen

SEND A MESSAGETo launch a messenger to type texts

Place thumbs over the screen to show the user is ready to type texts

STREAM A VIDEO (TWO HANDS)To stream a video lastly seen

Place the phone on a surface, tilt and support with two hand from the back. Similar to taking picture/video gestures, clear the area in front of the screen.

TAKE A PHOTO/VIDEO (ONE HAND)To take a photo or videos

Hold the phone stable with one hand, either vertically or horizontally. Clear the area of the screen where the photo/video being taken is shown.

Place the phone on a surface, tilt and support with one hand from the back. Similar to taking picture/video gestures, clear the area in front of the screen.

ENTER THE PASSCODETo load the passcode screen

A thumb is placed on the screen after a long resting.

CONCEPT DEVELOPMENT

53


TAKE A PHOTO/VIDEO (TWO HANDS)To initiate a call

Hold the phone table with two hands, either vertically or horizontally. Clear the area of the screen where the photo/video being taken is shown.

REMOTE CONTROLTo start remote control the target product

Approaching toward.

Using a smartphone as a remote control is does not require pointing due to the technological advancement. However, gestures for using an IR remote control were thought valuable, for its strong implication for the functionality, accumulated through time.

RECORD VOICE/SOUNDTo initiate a recorder

Bring the microphone part of the phone close to the mouth, to make the voice heard better in a noisy situation.

The phone can be placed on a surface after initiating the recorder.

ORIENTING A MAPTo calibrate and launch a map application

Waving a phone in 8 figure.

Map, orientation apps often requires calibrating. If calibrating gesture can be used to start up those apps and the calibration, it could relieves annoyance of calibrating in midst of using such apps.

PLAY MUSICTo initiate a call

Pick up the phone in a resting state and plug an earphone in.

SNOOZE THE ALARM To snooze the alarm

Turning a phone could work to switch of an alarm.

Figure 4-4. Preliminary design of function-gesture mapping

CHAPTER 4.

54

METHODParticipants15 participants were recruited, including ten female and five male participants. The nationality of the participants was diverse; Danish, Norwegian, Swedish, Dutch, German, English, Czech, Hungarian, American, Taiwanese, Thai, and Korean. Most of them were in the twenties, only except for one participant being in early thirties. All of the participants belonged to the age group who are in possession of smartphone and actively using it (Income, 2016). It was also expected that their age group along with younger generation would be the first ones to try out and accept the Gesture control concept. The participants had a background in design or design research, either from their studies or working experiences.

ProcedureFirst of all, participants received an online survey, followed by an individual interview session.

In the online survey, the participants were given a description explaining the basic idea of the concept. Thereafter, they were asked to evaluate gesture and function pairs, concerning a given task. Each gestures were presented with pictures, or videos if necessary. The evaluation criteria for the assessment were intuitiveness, ease of use, and social awkwardness, which were to be rated on a scale from 1 (very poor)-5 (very good). The description for each of the criteria was provided as shown below.

IntuitivenessVery artificial (1) - Very intuitive (5) Being intuitive means that a gesture and function have a strong association. The meaning matches well naturally, rather than random or artificial.

Ease of useVery hard (1) - Very easy (5) Being easy to use means that the gesture designed is simple, and can be performed engaging fewer body parts (e.g. using one hand instead of using two hands) with fewer muscle movements.

Social awkwardnessVery awkward (1) - Very natural (5) Being ‘not’ socially awkward means that it is very natural, subtle and not awkward to perform a certain gesture in the public, or even alone.

The tasks included: ‘Make a call’, ‘Write a message’, ‘Take a photo/video’, ‘Record a voice/sound’, ‘Play music’, ‘Stream a video’, ‘Enter a passcode’, ‘Remote control’, ‘Open a map’, ‘Turn off alarm/timer’. The ‘Take a photo/video’, and ‘Stream a video’, included two different gestures each, as the dominance between the two

gestures was unclear, thus selecting one of them was not done in the previous stage. After the survey, the results were discussed with the participants through an interview.

4.1.4 USER TEST: EVALUATION OF PRELIMINARILY SELECTED GESTURES

The preliminarily selected gestures were evaluated through a user test for the assessment. The main objective of this research was to verify if each of the gestures had the potential to be implemented as a new gesture concept for smartphone.

CONCEPT DEVELOPMENT

USER RESEARCH - Gesture Control concept forSmartphones* Required

INTRODUCTION

This research is about a smartphone, that can be controlled by a set of gestures.Thank you for participating in our survey.Your feedback is important. If you have any questions, please let me know by contacting to: [email protected] Please click ''continue'' to proceed.

CONCEPTIn this research, you will be asked to evaluate various gestures in accordance with a specific functionality.

When using a smartphone, some actions that you make have a very strong connection with the function that we are using. For example, when you are answering a call, it is very likely that you are bringing your close to your face, so the speaker and microphone would be positioned near your ears and mouth respectively.

As such, when such a gesture is performed, this new smartphone, controlled by gesture, knows that you are going to receive the call, and initiate the call without clicking any buttons on the screen. Similarly, you can also initiate a call only by picking up your phone and bringing to your face, then say a name in your contact list.

Likewise, some other gestures also can be associated with different functionalities of a smart phone. From the next page, you will be given with 10 functionalities of a smart phone, each of them related with one or two gestures as a trigger. Please evaluate each of the gestures based on 3 criteria; Intuitiveness, Ease of use, and Social awkwardness.

[1] Intuitiveness: Being intuitive means that a gesture, and a functionality have a strong association. The meaning matches well naturally, rather than random or artificial.[2] Ease of use: Being easy to use means that the gesture designed is simple, and can be performed engaging less body parts (ex.one hand) with less muscle movements. [3] Social awkwardness: Being 'not' socially awkward means that it is very natural, subtle and not awkward to perform a certain gesture in the public, or even alone. Please click ''continue'' to proceed.

MAKE CALLThis gesture is designed to initiate ''Making a call'' function.

1. *Please evaluate the gesture based on the criteria shown below. How do you feel about'brining a phone near your ear' to 'make a call'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)

Intuitiveness (Very artificial -Very intuitive)Ease of Use (Very hard -Very easy)Social awkwardness (Veryawkward - Very natural)

SEND A MESSAGETwo gestures shown below are designed to initiate ''Send a (text) message'' function.

Activation gesture: Bring a phone near your ear (and lips)

http://youtube.com/watch?v=Uzr_5Q9N52c

Activation gesture: Placing two thumbs on a screen

55


Figure 4-5. Evaluation results of the preliminarily selected gestures

REMOTE CONTROL ORIENTING A MAP

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE

SOCIAL AWKWARDNESS 4.2

4.3

4.2

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


3.4

3.0

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


3.3

3.4

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


3.3

3.2

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


3.8

3.6

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


3.6

3.6

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


2.9

2.6

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


2.6

2.8

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


3.4

3.9

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


3.8

4.0

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


2.2

1.9

0 1 2 3 4 5

INTUITIVENESS

EASE OF USE


3.9

4.3

TAKE A PHOTO/VIDEO (TWO HANDS)

MAKE A CALL

STREAM A VIDEO (ONE HAND)

RECORD VOICE/SOUND

SEND A MESSAGE

STREAM A VIDEO (TWO HANDS)

PLAY MUSIC

TAKE A PHOTO/VIDEO (ONE HAND)

ENTER THE PASSCODE

SNOOZE THE ALARM

CHAPTER 4.

56

RESULT The overview of the study results is shown in Figure 4-5. The average scores for each of the criteria are shown. The original results and interview transcripts can be found in Appendix H. Preliminary Gesture Evaluation Study.

In general, most of the gestures were evaluated intuitive, easy to use and not awkward. Although the degree of intuitiveness differed among the gestures, all of them were evaluated to be understandable. Few exceptions were: orienting a map, streaming a video, and taking a picture/video with two hands. The participants thought those gestures were not natural, evoking awkwardness, or not very easy to perform.

ANALYSISThe user study suggests factors that influenced the assessment of intuitiveness, ease of use, and social awkwardness. Moreover, the analysis of such factors provided design implications.

INTUITIVENESS• Integration with the original task actions

The more similar a gesture was to the original actions, the more intuitive it was evaluated. Of course, such a gesture has to be symbolic enough to represent a task rather than simply being a part of the original actions performed to achieve a task. Resemblance(relevance) with personal ritual.

• Personal ritual influenced the intuitiveness The gestures for recordings, snoozing, or turning off are not a part of the original action, but they were perceived intuitive as they resemble personal rituals. In the meanwhile, the evaluation of intuitiveness of ‘Write a message’ gesture was not consistent. While some participants who would type with two hands thought the gesture was intuitive, the other participants, who would type only with one hand, felt that the gesture was not intuitive for them, although they could understand why the gesture was designed as such.

• Resemblance with customs/physical world‘Remote control’ gesture was evaluated to be intuitive, even though smartphone offers a remote control functions over Wi-Fi, which does not require pointing. It was considered because of the old conventional way of using Infrared (IR) remote control. Some participants mentioned it would be nice to have a remote control function on a smartphone that could recognize a target device by just pointing at it instead of finding and launching an application.

• Mapping with functional componentsRecording gesture, to bring the bottom part (where the microphone component is) closer to the mouth was evaluated poorly by people who were not very aware of the location of the microphone component. For those people, the gesture was

not considered intuitive, since they did not find any meaningful connection. Once they know the location of the microphone, intuitiveness was evaluated positively.

EASE OF USE• Engagement of less hands

In general, use of one hand was considered easier than using two hands. However, using a camera was exceptional since many participants thought holding a smartphone with one hand was not stable enough to take a good picture.

• A shortcut by removing stepsRemoving steps, shortening time were considered to be one of the biggest benefits. The Gesture control can be complimentary, of an alternative way to assess certain functions. So when it becomes an extra work, more burdening than on-screen controls, it seems like that it loses its charm.

• Vague or confusing gesture affordanceDifferent gestures were suggested to stream a video and vice versa. Firstly, some participants mentioned the possibility of using other gestures for the ‘Stream a video’, such as holding the phone with one hand in the air. Also, some other participants associated the ‘Stream a video’ gesture itself (to place the phone stable on a surface) with different functions such as: browsing a web page or recording a video. Weak gestural affordance seems to be a reason behind the results mentioned above. Such uncertainty in gesture affordance also raised concerns about the feasibility of gesture detection.

• Doubt for gesture recognitionSome participants expected some of the subtle gestures hard to be recognized by Smartphone. It influenced them to give low scores for Ease of use criterion. They thought gestures with poor recognition will make them confused and hard to use the system.

SOCIAL AWKWARDNESS• Artificial/defined gesture

Drawing 8 horizontally for orienting a map evoked social awkwardness, for both people who already know this gesture, and who did not know this gesture. For the former group, who knew the ‘drawing 8’ gesture, the gesture still evoked social awkwardness. Drawing 8 horizontally for orienting a map engages a rather big movement, and it is noticeable in public. The idea of being noticed to use a map function in public, which probably indicates that the user is lost, evokes negative emotion to the user.

• Big movements drawing attentionFor the latter group, who did not know the ‘drawing 8’ gesture, it just felt awkward to perform such a gesture because they were not used to it at all. Furthermore, they did not understand why such a gesture is mapped with a calibration function.

CONCEPT DEVELOPMENT

57


• Nature of tasks drawing attentionTasks such as taking a picture or recording sound draw attention from others, not only because of the actions but also the characteristic of the tasks. Performing such tasks in public can be seen weird, and this can evoke social awkwardness. This was however seen irrelevant to design implication.

• Unfamiliar/infrequently used tasksGestures that were bound to some tasks (e.g. recording, orienting a map) that are not used frequently also raised awkwardness to some of the participants. The awkwardness, in this case, was not because of the gestures, but performing the tasks made the participants feel strange.

DISCUSSIONThere were some misunderstandings of the participants during the survey, and this influenced the assessment. This was mainly due to the wordings to describe some of the functions. It was later clarified during the following interviews.

Sending a message‘Sending a message’ function was defined to be triggered by “placing thumbs over the screen”. This brought confusion to some of the participants. Some participants misunderstood this function. Instead of activating a messenger app to send a message, it was understood actually to trigger ‘sending’ a message. The gesture was misunderstood few times. Two participants thought the gesture was to press the screen with two thumbs instead of hovering on the screen. The use of ‘launch the messenger app’ and ‘hovering over the screen’ are suggested.

Entering a passcodeSome participants understood this function similar to touchID (entering a passcode with a fingerprint) instead of light up the lock screen. This will have to be clarified to light up the screen, or wake up the device from a resting state.

Play musicInstead of ‘Play music’, ‘Launch the music app’ is suggested for the function. The users preferred to have control over the music app instead of automatic play.

Record voice/soundThis raised some confusion to the participants, whether the recording starts immediately or a recorder application is launched. The title is suggested to be changed to ‘Launch the recorder’.

CONCLUSIONIn general, gesture control for a task should be derived from original actions required to perform the task. However, the gestures should have clear gesture affordance, and be unique enough to be distinguished from other gestures. The design of the gestures can be inspired from the customs/physical world, or personal ritual.

Using one hand is easier in general than two hands. However, the use of only one hand simplifies gestures and making it more general. This can create confusion by having less gestural affordance, and conflict with other tasks. Furthermore, gesturing with two hands is not considered very difficult, but only relatively when compared with one-hand gestures. The use of one hand and two hands can be balanced depending on the situation.

Creating a shortcut is a significant design intention. The overall experience should not demand additional physical or mental effort. Additionally, characteristic of a task, the frequency of a task can be considered to determine how valuable it is to create a shortcut for such a function.

Social awkwardness can be avoided with the use of natural gestures. One is to use a conventional gesture that is natural to use and to be seen thus draws less attention. The other strategy is to use subtle gesture, to be seen less in the first place. It should be kept it mind that the subtlety and distinctiveness of the gestures should be balanced, in order the system to be technologically feasible.

Lastly, some of the names of the functions need to be reconsidered as it brought some confusions to the participants.

CHAPTER 4.

58

The main function of the concept is to trigger smartphone functions with gestures.

This function includes three sub-functions; gesture library, gesture recognition, and application (function) trigger. Solutions for each of the sub-functions were generated separately in an analytical and systematic way using Morphological chart (Roozenburg & Eekels, 1995). The solutions were evaluated with the users, then the final concept was designed. See Figure 4-6 for the Morpholocial chart.

4.2.1 GESTURE LIBRARY

Two solutions were thought for the gesture library; flexible library and personalizable library.

The gesture library refers to a set of gestures that are used to trigger a function. In order to be intuitive and easy to use, these gestures should be designed conventionally. However, being conventional can be subjective as the users showed personal preferences in the previous user test. Thus, two different solutions were thought, the flexible library offering more gestures to meet more user needs, and the personalizable library offering customizability to the users.

4.2.ITERATION 2: WIREFRAME CONCEPT DEVELOPMENT

Solution #1-1 Flexible libraryFlexible gesture library maps one function with multiple gestures. Performing one of the gestures can trigger the corresponding action. The examples are shown in Figure 4-7. This solution aimed to integrate as many gestures as possible, so the gesture library can be flexible to more users, and scenarios.

CONCEPT DEVELOPMENT

Figure 4-6. Morphological chart used during the design process

Sub-functions Solutions

FINAL CONCEPT

#1FlexibleLibrary

Related technologies to be discussed in Chapter 5.

#2Personalizable

Library

#1Customizable

Settings

#2Selectable

List

#3Situational

Modes

1. GestureLibrary

GestureRecognition

2. Application(Function)

Trigger

59


Figure 4-7. Wireframe design for the flexible library concept

1

5

9

2

6

10

3

7

11 13

4

8

12 14

CHAPTER 4.

60

Solution #1-2 Personalizable libraryThe personalizable library provides an option to select one gesture for one function based on the user preference, as shown in Figure 4-8. This solution offers customizability in the gesture library. This aims to enhance the understanding of the users regarding the gesture library by giving them a chance to flip through the library when selecting the gestures. Furthermore, using one gesture for one function can be thought easier to use, as there will be fewer options to remember.

Figure 4-8. Wireframe design for the personalizable library concept

1 5

9

2 6

10

3

7 11

4

8 12

CONCEPT DEVELOPMENT

61


4.2.2 Haptic feedback for gesture recognitionWhen a gesture is performed and recognized by the smartphone, haptic feedback (vibration) is given to the user.

When it comes to product interaction, Feedback is essential to indicate the successfulness or unsuccessfulness of an action (Norman, 2013). Visual feedback is crucial in Smartphone, in which main interaction happens on the screen. However, the benefit of the design concept to eliminate steps to launch a function sacrifices this chance to check such visual feedback. In order to deal with this conflict, another way of giving feedback for the control was sought.

Haptic communication recreates the sense of touch by applying forces, vibrations, or motions to the user (Goldstein, 2010). The

short and weak vibration was considered as good feedback to inform successful gesture recognition, for the execution of most of the functionalities. The property of being short and weak was considered not to be confused with the vibration already implemented in Smartphone. Considering the subtlety of the design concept, it is more convincing that such a haptic feedback should advocate the same characteristic. It is also thought to create less annoyance that could be potentially caused by strong haptic feedback.

Snooze/Dismiss a call, Snooze/Turn off alarm were excluded as they already result in sound or vibration change, which provides instant feedback to users. Launch the music app was also excluded, as it necessarily requires the user to look at the screen while plugging in earphones, and also to select music.

CHAPTER 4.

62

4.2.3. APPLICATION TRIGGER SCENARIO

Three different solutions were designed for the application trigger; to provide customizable trigger settings (Figure 4-9), a selectable list on trigger (Figure 4-10), and situational modes (Figure 4-11).

Some applications have the same core function. For example, iPhone camera application, Instagram, and Snapchat share exactly the same essential function to launch a camera to take a picture/video. It is inevitable that one gesture can be associated with more than one application. If a gesture triggers an application that was not intended, this will create a very frustrating experience. In order to design a smart system that predicts user needs more precisely, different solutions were suggested.

Solution #2-1 Customizable trigger settingsThis solution asks the user to set one application for one gesture in the settings. Whenever a trigger gesture is recognized, the set application will be launched.

Figure 4-9. Wireframe design for the solution: customizable trigger settings

A-1

B-3

A-2

B-4

A-3

B-1

A-4

B-2

CONCEPT DEVELOPMENT

63


CHAPTER 4.

64

Solution #2-3 Situational modesThe system triggers one application depending on a situation. This solution provides a setting for different situational modes, such as default, work, meeting, home, active, etc (Figure 4-11:2). The situation can be set by the use easily, similar to the setting of the airplane mode (Figure 4-11:1). For each situation, the user can choose one app to trigger in the setting.

Solution #2-2 Selectable list on triggerA recognized gesture results in a list of associated applications. It allows the user to select the desired application to be launched from the list. Different selection method was suggested. One was to use traditional touch (Figure 4-10: A-3), and the other was to tilt or shake a phone to the designated direction (Figure 4-10: B-3).

CONCEPT DEVELOPMENT

65


Figure 4-10. Wireframe design for the solution: Selectable list on trigger

Figure 4-11. Wireframe design for the solution: Situational modes

A-1 A-2 A-3 A-4

1 2 3 4

B-3 B-4B-1 B-2

7 85 6

CHAPTER 4.

66

OBJECTIVES

This user test aimed to see the user preference on the previous solutions proposed for the concept. It covered the assessment of gesture library, and the application trigger scenario.

METHODS

ParticipantsFive participants were recruited for the user research.

All of the participants had an education or working background in User experience design. The age varied from 22 to 27, which did not necessarily represent the general population, but their experiences in design industry were considered to compensate it, as they are familiar with the empathetic thinking process, not only considering their personal preferences but also considering other populations.

ProceduresThe contents were built in Wireframe, to evoke discussion mainly on the interactions and functionality of the concepts, instead of focusing on minor application design elements. The concepts were delivered to the participants in a form of paper prototype (Figure 4-12). Paper prototyping was chosen because it provides a simple, cheap and quick way of introducing different interim concepts. Furthermore, a polished software prototype gives an impression that the concepts are already concrete, which can restrict participants to share their opinions straightforward (Krug, 2013).

The participants were given with a real smartphone along with the paper prototype. The participants could try out the gestures while interacting the real smartphone. This left much fewer imaginations but more realistic ideation, providing information about the size, hardware interface of a real smartphone.

Figure 4-12. User test setup and the materials used

4.2.5 USER TEST: CONCEPT SELECTION

The second user test was conducted. It aimed to see the preference of the users regarding different solutions and evaluate different design solutions for the gesture library, and application trigger scenario.

CONCEPT DEVELOPMENT

67


RESULTS

Gesture libraryThe participants liked the combination of the two gesture libraries instead of choosing either of the two. They thought both directions were necessary. See Table 4-2 for the overview of the results.

The participants wanted the gesture library to be flexible, especially for ‘Activate the camera’ function. They also wanted some customizability. They wanted to turn off functions on the personal preference. For example, one participant mentioned that he will never use a recording function on the phone, and it will feel more secure if the gesture trigger can be turned off for the function.

Application trigger scenarioThe overview of the results can be found in Table 4-3.

All of the participants agreed with the necessity of the settings, as long as it is made simple and easy to use, especially regarding first-time use.

Most of the participants preferred the first solution, to assign one particular app to be triggered with one gesture. It was evaluated to be the simplest and easiest. The participants did not prefer

to deal with more settings after going through a whole gesture library. Although the necessity of this step was acknowledged by the participants, they still felt it was overwhelming. It seems that the presentation, and interaction of the settings can be designed better to create a different experience. The participants showed different opinions when it came to the message function. Except for one participant, the rest of the participants answered that they use different message apps daily basis, and the preference is indecisive. Some participants mentioned that they would like to be able to select one app from a suggested list (second direction), the others still mentioned that they would like to have only one app connected to gesture while the other apps can be selected by touch - in a classical way. This leaves a challenge for a designer to decide which direction will be more suitable and beneficial for the concept.

The third solution, to provide different modes, the basic idea was appreciated by the most of the participants. However, it was considered premature to be introduced and burdensome to make settings. Some participants, who thought the settings were overwhelming to make changes one by one, wanted a pattern-learning feature. Some other participants wanted such settings to be introduced in the latter stage, for an advanced user or as a more advanced feature to be developed in the future, when this gesture control concept for a smartphone is widely accepted by the market.

Table 4-3. Summary of the research results for the application trigger scenarioTable 4-2. Summary of the research results for the gesture library

#2-1 #2-2 #2-3 Remarks

Participant 1 O ∆ ∆ #2-3 definitely not a first-time setting, super user option.

Participant 2 ∆ O ∆ #2-1 depends on the type of apps.#2-3 only if made easy, bound to my calendar, etc.

Participant 3 O ∆ X #2-2 only for messaging app, not all the time. #2-3 each mode doesn’t have clear preference.

Participant 4 O X ∆ #2-3 the next feature to launch.

Participant 5 O X ∆ #2-3 advanced feature to consider in the future.

#1-1 #1-2

Participant 1 O O

Participant 2 O O

Participant 3 O O

Participant 4 O O

Participant 5 O O

O : Definitely preferred ∆ : Somewhat preferred X : Not preferred

O : Definitely preferred ∆ : Somewhat preferred

X : Not preferred

CHAPTER 4.

68

ANALYSIS AND DISCUSSIONFlexible and customizable gesture libraryWhen it comes to a gesture library, the gesture library has to be flexible enough, but users also have to be given with some customizability. Flexibility contributes to better satisfy different preferences of the users on using gestures. Customizability can be an opportunity for the users to learn the system better while going through the whole library. It is also expected to protect accidental recognition of gestures by letting the users disable certain gestures perceived not intuitive enough. This will increase credibility in the system, which will enhance the user experience. In short, in combination of two different solutions, to have a flexible gesture library with a possibility of user customization seems to be the most attractive direction.

Application launch settings: Balance between simplicity and perfectionThe participants considered simplicity to be very important. A simple, easy and straightforward system will be more appreciated than an advanced but complicated system. It was also found that they would feel frustrated by going through many settings and options even though it is needed for a perfectly working system.

Snoozing and turning offThere were few discussions concerning gestures for snoozing and turning off functions for both of the phone call and alarm function. During the user research, snoozing function and turning off function were introduced with ‘flip over’, ‘covering up the (whole) screen for a second’, respectively. Technically, the ‘flip over’ gesture requires more movements, and motor skills than just putting a hand on a smartphone screen. However, ‘flip over’ was perceived as demanding less effort than ‘covering for a second’ by the participants. ‘Flip over’ gesture being more fast and instant seems to be the reason for this.

Technological feasibilityIssues regarding technological feasibility were addressed. Some of the gestures raised concerns in terms of their technological feasibility rather than intuitiveness or ease of use. Some of the participants questioned the reliability of the gesture with subtle details such as hovering a thumb on a shutter button position, hovering two thumbs over the screen.

Context awarenessSome of the participants mentioned that the context of the gesture control, or the state of a smartphone in another word, has to be considered, to prevent accidental trigger of a function. The state should consider different information such as the position, the location of the phone, or a foreground application (which is an application that currently has the input focus). When taking out a smartphone from a pocket to answer a call, the system has to know the screen covered by hand does not mean rejecting a call in this case, by recognizing its previous position or location. Using the same gesture for a phone call and alarm already provides a good example to address the importance of recognizing the foreground app. As such, more cases can be explored to design a less faulty, more

credible system.

Pattern learning systemPattern learning was mentioned during the research. Similar to Siri’s app suggestion on iPhones, based on a user’s pattern recognized from calendars, emails, time and location, the system might be able to suggest an app that is suitable under a particular circumstance. However, it is concluded to leave out mainly for two reasons. First of all, reading a pattern is a complicated task that requires a lot of data to be processed. Thus, it was thought to be out of the scope of this project. Secondly, if the system misinterprets the pattern and triggers an unwanted result, it could actually create more annoyance rather than being helpful. It will be different from the Siri suggestion that only makes suggestions and triggers no actual function.

CONCEPT DEVELOPMENT

69


CONCLUSIONThe second iteration cycle leaves implications for the design of the gesture library.

When it comes to a gesture library, users have to be given with not only a flexibility but also customizability. Functions that one user can use multiple gestures depending on the situation (e.g. Activate the camera) should include multiple gestures. The user should be able to turn off unwanted functions as well. This will contribute the performance of the system by reducing accidental trigger.

Application setting should be designed simple. In a nutshell, the simplest and the easiest way to do so is a one-to-one mapping of gesture and application. This could result in annoyance to some users, thus other options (Solution #2-2) can be implemented partially. The good example is a message function, as the users frequently have multiple applications in use, and the dominance is not clear.

CHAPTER 4.

70

In the initial smartphone gesture control concept, the remote control function enables remote controlling of a target object by pointing at it. This function was developed further to enable gesture controlling at home while using a smartphone as a medium. Using a smartphone might not be as empowering as gesture control with human hands and body. Nonetheless, smartphones are very easy to be accessed any anytime, and it is always kept very close. Furthermore, technologies that are already realized in Smartphones can eventually benefit the feasibility of the concept.

When it comes to the objects to control, main domains for home automation such as HVAC, lighting, Audio-visual, Shading, Security, Intercom, etc. were considered, as discussed earlier in the research phase of this project. Eventually, thermostat, lighting, speakers, blinds, and curtains were chosen considering familiarity, usage, and context of home living.

4.3.1. BASIC INTERACTION MODEL

A new concept was built considering intuitiveness, ease of use, and social awkwardness, based on the same principle as the smartphone Gesture control. The main objective was to come up with a very natural gesture library, so it hardly puts any extra physical or mental effort to the user. Firstly, tasks performed when interacting directly (without a remote control) with a certain subject in real life were investigated, and then a basic interaction model was created. This Analogies & metaphors (Boeijen, Daalhuizen, Zijlstra & Schoor, 2010) method was used for the inspiration at first. In the latter process, it was thought that the similarity of the interaction model with the real life tasks would make the Gesture control to be perceived more natural, as it can reduce a learning curve by making use of the experience and existing knowledge, instead of forcing to learn a very new set of controls.

The real life interaction model was thought to: Think – Approach (walk toward) – Touch – Control, as seen as in Figure 4-14. In real life, interaction with a person and a lamp can be described as below. The person initially thinks about controlling the lamp. Then the person will approach to the lamp, without a remote control, or any separate physical switch. The person eventually puts his/her hands on the lamp and perform a certain action to accomplish a task, pulling a string, for example.

Based on the interaction model above, the basic model for Gesture control using a smartphone was created to: Think - Point - Push - Control, as seen in Figure 4-15. Walking toward and touching the object were translated into pointing and approaching. The approaching action was designed to be ‘pushing forward’, as it is evident that smartphone cannot perform walking toward.

When pointing an object with a smartphone, the smartphone attempts to make an initial connection with the object (initial connection). Once the initial connection is established, the connection can be confirmed by pushing the phone forward, and the remote control function is fully activated. After this activation, the object is ready to receive functional gesture commands.

ITERATION 3: SMARTPHONE REMOTE CONTROL

4.3.

CONCEPT DEVELOPMENT

Figure 4-14. Real life interaction model

Figure 4-15. Proposed interaction model for the remote control

71


4.3.2. GESTURE DESIGN

GesturesWhile the basic model focused more on the establishment of the initial connection between a smartphone and target (activation), gestures for controlling different products were explored after the basic framework was set. Two different types of tasks were considered when designing such action gestures. The first one is the switch function, to turn an object on and off, and the second one is to adjust the object’s property such as brightness, volume, or location.

Initially, various ways to control things in real life were investigated for the inspiration. Various examples such as a knob, string, slide switch, push button, toggle button, etc. were found, and then various metaphors were found from those examples.

Interface Metaphor

Toggle switch Up-down vertical movement

String switch Up-down

Push switch Push forward, or push down

Slider switch Up-down or Left to right sliding

Knob/dial Clockwise, or counter-clockwise circular motion

Table 4-4. Real life interface and metaphors

However, using a different metaphor for each of the products was thought to be confusing, thus, it was thought that it would be more beneficial if the gestures could be designed more universal.

As a result, Gesture control for lighting, speaker, blinds turned out similar, as all of them have a tendency for vertical movements. Gestures for controlling curtains are a bit different since the curtains move horizontally. For the thermostat, even though the interface of the thermostat could vary, either vertical, horizontal, or circular, it was thought that it would be very easy to perceive that increasing and decreasing temperature functions are mapped with moving up and down, showing a clear metaphor of increase and decrease.

Basically, the gestures were designed to be as simple as possible to be performed easily, and also to be easy to remember. The combination with haptic feedback was also considered. The circular motion was not preferred for this since fragmented interaction would give more possibility to be integrated with the haptic feedback. Detailed scenarios can be found in the next page (Figure 4-16. Figure 4-17).

CHAPTER 4.

72

BASIC SCENARIO

DUPLICATION SCENARIO

CONCEPT DEVELOPMENT

ACTION - SWITCH: Tilt up

ACTIVATION CONFIRMATION: Push forward to confirm the connection

INITIAL CONNECTION: Aim and point the lamp to establish the initial connection

ACTION - SWITCH: and down to turn on/off the activated device

ACTION - ADJUSTMENT: Move upwards to brighten up

ACTION - ADJUSTMENT: Move downwards to dim down

ACTION - DUPLICATION: Press screen when the connection is still valid

ACTION - DUPLICATION: Point at the target device while pressing down the screen

ACTION - DUPLICATION: Remove the thumb to confirm the input

ACTION - DUPLICATION: Press screen again to duplicate settings again

ACTION - DUPLICATION: Drag and drop to the target device as described before


Figure 4-16. Basic Remote control function

Figure 4-17. Duplication function of the Remote control function

73


Haptic feedbackHaptic feedback using vibration is considered important, especially in this case when visual feedback is not necessarily given nor considered important due to the characteristic of Gesture control. Three different types of vibration were designed to serve different purposes.

Firstly, constant and subtle vibration is given after the initial activation, when a smartphone is aimed at a certain object to control. Thereafter, at the moment when the user confirms the activation by pushing the phone forward, the intensity of the vibration increases to indicate the successful activation, and the object is ready to receive input. When any input is given to perform a function on the object, a strong and short accent vibration will be given to indicate the success of the control, then simply going back to the medium vibration state until all the connection is lost, by putting the smartphone down, or by pressing its sleep button.

Figure 4-18. Vibrations considered for the haptic feedback

Visual feedback on the targetBasically, no visual feedback is given to the smartphone, as it only remains as a very subtle medium to deliver gestural inputs. Instead, the Gesture control concept is expected to establish a strong connection between the user and the object to be controlled. It is very likely that users are looking at the object when performing controlling gestures. Thus, if any visual feedback is necessary, it has to be given to the object where the user is looking at. As a result, the use of an additional LED was added, to give feedback for initial activation and the activation confirmation. A very similar example can be found in daily life, which is a small LED light of a TV, blinking in accordance with a remote control.

Flexibility in SpeedDifferent speed of performing gestures affects the speed of the actions.

CONSTANT SUBTLE VIBRATIONUsed to indicate initial activation after aiming

MEDIUM VIBRATION

ACCENT VIBRATIONUsed to indicate the excuation of a certain function

CHAPTER 4.

74

User ScenarioLightingActivation and activation confirmation is established by pointing and pushing. Pulling up shortly enables switching on or off, depending on the status of the lamp. Brightness can be adjusted by moving up and down (Figure 4-19).

Figure 4-19. User scenario for the light control

SpeakerGesture control for a speaker is done exactly the same as the lighting. Volume can be adjusted by moving up and down (Figure 4-20).

Figure 4-20. User scenario for the speaker control

CONCEPT DEVELOPMENT

ACTION - ADJUSTMENT: Drag upward to brighten

AIM: Pointing for initial activation

APPROACH: Push forward for activation confirmation

ACTION - BASIC: Pull up/down to turn on/off

ACTION - ADJUSTMENT: Drag upward to brighten

ACTION - ADJUSTMENT: Drag upward to turn up the volume



ACTION - BASIC: Pull up/down to turn on/off

ACTION - ADJUSTMENT: Drag upward to turn up the volume

75


ACTION - DROP: Connection lost


ACTION - ADJUSTMENT: Drag downward to dim

ACTION - ADJUSTMENT: Drag downward to dim



ACTION - ADJUSTMENT: Drag downward to turn down the volume

ACTION - ADJUSTMENT: Drag downward to turn down the volume

CHAPTER 4.

76

Blinds/curtainsGesture control for horizontal blinds (Figure 4-21) is as same as the lighting, and speaker. For controlling vertical blinds/curtains (Figure 4-22), left and right horizontal movement is used, instead of up and down, considering the movement of the curtains in real life.

Figure 4-21. User scenario for the horizontal/roller blinds control

Figure 4-22. User scenario for the vertical blinds/curtains control

ACTION - ADJUSTMENT: Drag upward to roll up the blinds

BLINDS



ACTION - BASIC: Pull up(down) to fully roll up(down) the blinds

ACTION - ADJUSTMENT: Drag upward to roll up the blinds

ACTION - ADJUSTMENT: Drag to the left to open the curtains to the left



ACTION - BASIC: Swipe to left (or right) to fully open the curtains

ACTION - ADJUSTMENT: Drag to the left to open the curtains to the left

CONCEPT DEVELOPMENT

77




ACTION - ADJUSTMENT: Drag downward to roll down the blinds

ACTION - ADJUSTMENT: Drag downward to roll down the blinds



ACTION - ADJUSTMENT: Drag to the right to open the curtains to the right

ACTION - ADJUSTMENT: Drag to the right to open the curtains to the right

CHAPTER 4.

78

ThermostatThermostat works as same as the lighting, speaker, and blinds (Figure 4-23).

Figure 4-23. User scenario for the thermostat control

Duplication gesturesWhen it comes to Lighting and speakers, it is likely that there are multiple numbers of lamps and speakers in a certain space at home. A shortcut for duplicating (or copy and pasting) the same setting from one appliance to another were ideated. Mainly inspired by drag-and-drop, it was adapted more suitable to a smartphone. It is thought to be useful when a user wants to turn on multiple lights at the same time, saving effort and time. Also, turning on two or more speakers at the same time can be done simply using this gesture. See Figure 4-24 for the interaction scenario when applied to a lighting system.

Figure 4-24. User scenario for the duplication function



ACTION - BASIC: Pull up(down) to increase the temperature

ACTION - ADJUSTMENT: Drag up to constantly increase the temperature

ACTION - ADJUSTMENT: Drag up to constantly increase the temperature

ACTION - DUPLICATION: Press screen when the connection is still valid

ACTION - DUPLICATION: Point at the target device while pressing the screen

ACTION - DUPLICATION: Remove the thumb to confirm the input

ACTION - DUPLICATION: Press screen again to duplicate settings again

CONCEPT DEVELOPMENT

79





ACTION - ADJUSTMENT: Drag down to constantly decrease the temperature

ACTION - ADJUSTMENT: Drag down to constantly decrease the temperature


FINAL DESIGNThe final design, “Smartphone Controlled Home Environments” is introduced in this chapter. The concept description, features, user scenarios and expected benefits are illustrated along with its technological feasibility provided by the technology research.

5

82

CHAPTER 5.

CONCEPT5.1.

FINAL DESIGN

WAKE UP THE DEVICE

MAKE A CALL

ANSWER A CALL

SNOOZE A CALL

REJECT A CALL

LAUNCH THE MESSENGER

ACTIVE THE CAMERA

SNOOZE THE ALARM

TURN OFF THE ALARM

LAUNCH THE RECORDER

LAUNCH THE MUSIC APP

START REMOTE CONTROL

“

”

The final concept is an interaction concept for a smartphone control, including a remote control function for home environments. It includes twelve smartphone built-in functions which are triggered by gestures. One of the function extends the usage of a smartphone in home environments, as shown in Figure 5.1.

The twelve functions are; ‘Wake up the device’, ‘Make a call’, ‘Answer a call’, ‘Snooze a call’, ’Reject a call’,

‘Launch the messenger’, ‘Active the camera’, ‘Snooze the alarm’, ‘Turn off the alarm’, ‘Launch the recorder’, ‘Launch the music app’, and ‘Start remote control’.

The concept is represented in the original smartphone UI. It was done on the assumption that the concept is included in the settings menu of the smartphone. The name of the functions and the descriptions were modified to be more suitable to be presented to the potential users.

83


Figure 5-1. Illustration of the Remote control function of the final design

CHAPTER 5.

84

FINAL DESIGN

WAKE UP THE DEVICE

‘Wake up the device’ function lights up the screen to load a home screen or a lock screen, when the phone is picked up by the user from Sleep mode.

The gesture is described with detailed motion graphics

The description combining the function and gesture is provided

User can turn off unwanted functions

GESTUREILLUSTRATION

DESCRIPTION

CUSTOMIZABILITY

‘Make a call’ function is to make a call by bringing the phone to an ear and say a contact name.

MAKE A CALL

5.2.FEATURES

TWELVE FEATURES

85


Figure 5-2. Final gesture-function mapping for the Smartphone interaction

‘Snooze a call’ function is to snooze an incoming call by flipping over the phone, whichever the side the phone was facing up.

‘Reject a call‘ function is to reject an incoming call by covering up the screen with a hand for few seconds.

‘Answer a call’ function is to answer a call by brining the phone to an ear, when the user is getting an incoming call.

ANSWER A CALL SNOOZE A CALL REJECT A CALL

CHAPTER 5.

86

FINAL DESIGN

LAUNCH THE MESSENGER ACTIVATE THE CAMERA (1)

‘Launch the messenger’ function is to launch a user defined messenger application, by hovering two thumbs over the screen. It normally launches a new message window, or it launches a current message window when getting a push notification for an incoming message.

‘Activate the camera’ function is to activate the camera, and one of the gesture commands is to hold the phone horizontally with two hands while stabilizing the phone.

TWELVE FEATURES

87



ACTIVATE THE CAMERA (2) ACTIVATE THE CAMERA (3) ACTIVATE THE CAMERA (4)

Holding the phone horizontally with one hand, while placing a thumb at a shutter button (volume button in iPhone) can trigger the ‘Activate the camera’ function.

Holding the phone vertically, while hovering a thumb on a shutter interface can trigger the ‘Activate the camera’ function as well.

‘Activate the camera’ function can be triggered by holding the phone horizontally with one hand,and hovering the thumb on a shutter interface.

CHAPTER 5.

88

FINAL DESIGN

LAUNCH THE RECORDER SNOOZE THE ALARM

TO LAUNCH THE RECORDER

‘Launch the recorder’ function is to launch a user-defined recording application when it is tilted and the bottom part (where the microphone is embedded) is held higher close to the face.

‘Snooze the alarm’ function is to snooze an ongoing alarm by flipping the phone over from whichever side, as same as the ‘Snooze a call’.

TWELVE FEATURES

89


TURN OF THE ALARM LAUNCH THE MUSIC APP START REMOTE CONTROL

‘Launch the music app’ is to launch a user-defined music application when earphones are plugged in.

‘Start remote control’ function is to control home appliances. The connection is established by pointing. Twitching, moving up/down (left/right) allows on/off and adjustment (e.g. intensity, scale, size) respectively.

Turn off the alarm’ is to turn off/cancel an ongoing alarm by covering up the screen with a hand for few seconds.


90

CHAPTER 5. FINAL DESIGN

COMPATIBILITY

The ‘Start remote control’ function offers control for various types of home appliances.

REMOTE CONTROL DETAILS

BLINDS/CURTAIN

LIGHTING

SPEAKER

THERMOSTAT

91


TURN ON/OFF ADJUSTMENT DUPLICATE

Drag-and-drop gesture copies the setting from one device to another. This function works between the same devices, such as two identical lamps, or the speakers.

Twitching gesture turns on or turns off a device, depending on the current state of the device.

Moving up/down or left/right adjusts the controllable measurement of a device.The direction depends on the type of the device (e.g. Up/down for volume, temperature, or horizontal blinds; Left/right for curtain and vertical blinds).


CHAPTER 5.

92

GESTURE – FUNCTION MAPPING

The final design includes twelve functions mapped with fifteen triggering gestures, and sub-gestures for the remote control (Figure 5-2, 5-3, 5-4, 5-5).

‘Wake up the device’ function lights up the phone screen when the phone is held up by the user. It removes one of the current steps to light up the phone screen to press a button. ‘Make/answer/snooze/reject a call’ functions are associated with a phone call function as the names already suggest. A call can be initiated by bring the phone close to an ear and say the name of a contact. When getting a call, the call can be answered by bring the phone close to an ear as well. To snooze an incoming call, the user can flip the phone over. To reject a call, the user can cover up the screen with a hand and wait for few seconds, without having to give a look and interact with the screen (Figure 5-2).

To write a text message, or to answer an incoming text message when getting a push notification, the user can launch a messenger application by making a standby gesture for typing which is to hover two thumbs over the screen. The ‘Activate the camera’ function can be performed by holding the phone stable for few seconds in specific postures concerning the location of certain fingers (Figure 5-3).

The recording function (‘Launch the recorder’ function) can be activated by brining the phone close to the mouth, while tilting the bottom part higher. The ‘Snooze the alarm’ and ‘Turn off the alarm’ functions works equivalent as the ‘Snooze a call’ and ‘Reject a call’ function, respectively. Plugging in earphones (an earphone) can launch the music application (‘Launch the music app’). Lastly, the ‘Start remote control function’ establishes a connection between the phone and a target product and enables remote controlling when the target product is aimed with the phone (Figure 5-4). The remote control allows turning on/off a device and the adjustment of a controllable measurement (Figure 5-5).

CUSTOMIZABLE TRIGGER SETTINGS

Firstly, the user can disable functions that they are not willing to use depending on the personal preference. This can be managed in the gesture library settings (Figure 5-2). This feature provides security by reducing accidental triggers and human errors.

The user can assign an application for few gestures that launches an application. Those functions are ‘Make a call’, ‘Launch the messenger’, ‘Activate the camera’, ‘Launch the recorder,’ and ‘Launch the music app’. When a gesture is performed for the first time, a pop-up screen will appear showing applications that share

Figure 5-7. Application trigger settings in the Gesture library settingsFigure 5-6. Application trigger setting for the First-time use

FINAL DESIGN

93


the same core functionality (e.g. iMessage, Facebook Messenger, and Whatsapp). The user can select an application that he/she would like to use as a default (Figure 5-6). Otherwise, the user can make or change the settings in the settings menu as well (Figure 5-7).

FLEXIBLE GESTURE LIBRARY

Two types of flexibilities are adapted in the interaction design concept.

Firstly, multiple gestures are linked with the activate the camera function (Figure 5-8). Four different gestures can trigger the function, and those derived from the original gestures currently used by people when taking a picture. All of the four gestures were decided to be introduce as people have different preference. The users can always use their favorite, or frequently used gesture to trigger the function.

Secondly, the users are given with a flexibility to disable some of the functions offered in the library. This decision was made to make the users more aware of the functions and reduce unwanted events to happen automatically.

REPRESENTATION OF GESTURE LIBRARY

The gestures in the gesture library are presented with detailed line drawings. The use of simplified icons was considered at first. They are trendy and beautiful, and they usually do a good job describing certain meanings. However, when it comes to describe gestures, especially when they are subtle, it was thought that detailed drawings would be much more effective. Some of gestures were animated for the easier and better communication.

HAPTIC FEEDBACK

Haptic feedback is given when a trigger gesture is performed and properly recognized. When a trigger gesture is recognized by the smartphone, it gives a short vibration to notify the confirmation. And then, the corresponding function is executed.

Figure 5-8. ‘Activate your camera’ function triggered by any of the four gestures (flexible gesture library)

CHAPTER 5.

94

5.3.USER SCENARIOS

BENEFITS

The benefits of the concept were readdressed during the user study, being a shortcut and reducing steps. If the system becomes complicated and fails to work as a shortcut, it will lose its charm. Another benefit was a reduced mental effort by less distraction. The increased possibility of not having to see and check the screen reduces distraction. As users can feel and see their own gestures, they are constantly reminded of the original interaction planned to do. It contributes the intuitive use of Smartphone.

The concept benefits the users by saving time, efforts and social awkwardness. Most of the features remove steps in the original smartphone interaction, thus saving time and effort. It also helps the users to focus less on the smartphone screen, which can benefit social awkwardness.

Wake up the device makes a Smartphone more responsive to the touch of the user. The ‘Make a call’, ‘Answer a call’ functions remove few steps to execute such functions. The ‘Snooze/Reject a call’, and ‘Snooze/turn off the alarm’ enable the users to preform such tasks even without a glance, and makes the users to focus more on their original activity. Also, this will bring back more physical, real-world interactions such as tapping the big snooze button in the morning.

The ‘Launch the messenger’ function removes few steps to write a text to someone, or to answer a text message, and it could create less social awkwardness while being with other people. Similarity, The ‘Active the camera’ function helps the user to take pictures not only intuitive, but also faster and less awkward. This may help to create meaningful interactions to capture precious moments while seeing over the screen, without looking through the screen.

The ‘Launch the recorder’ function eliminates the need of being lost in the screen looking for a function that are not used on a daily basis. Launch the music app also removes a step to launch the app. Lastly, the ‘Start remote control’ function provides more comfort and free control of home environments through the use of a smartphone.

FINAL DESIGN

Access functions quickly See over the screen, instead of see through the screen and seize the real moment

95


Interact with a smartphone, without a look Reduce time looking at the screen.

Figure 5-8. Examples of the benefits

CHAPTER 5.

96

5.4.TECHNOLOGY RESEARCH

5.4.1. SENSOR BASED GESTURE RECOGNITION

This topic discusses sensor based approaches to recognize smartphone posture changes, and hand gestures.

Position sensingPosition sensing enables a smartphone to assess its positional measurements in order to sense user’s gesture control.

Some motion sensors are already integrated into smartphones, such as a gyroscope, magnetometer (an element for the compass function), and accelerometer (Costello, 2016). These technologies can enable the detection of position changes of a mobile device whether it is in a sleep/rest state, moved or held in a certain position (T, 2014). This can enable features such as Wake up the device, Snooze a call, or Snooze the alarm. The technology also can help to detect the positional conditions for the activate the camera function.

Figure 5-9. Position sensing in iPhone

Proximity sensing Proximity sensing allows the detection of human body parts approaching to the smartphone.

Samsung galaxy smartphone series offers some Gesture control feature. Wake-on approach and answering a call by pick-up are one of the features requires face detection and hand detection. These type of features can be realized by the use of the capacitive proximity sensing (“Exclusive: Proximity Sensing in Mobile Phones”, 2014). The use of such technologies can help the realization of the Make a call function with a combination of the voice recognition, and answer a call.

While the detection of the proximity can detect either a face or hand is approaching, the capacitive measure can detect whether it is a face or a hand that is approaching (Baxter, 2000). The ‘Wake up the device’, ‘Answer a call’, ‘Reject a call’, and ‘Turn off the alarm’ functions can benefit from the same technology enabling the detection of the hand covering the screen. The capacitive sensor contributes to the detection hovering in the air, and this could help the recognition of ‘Launch the messenger’ function.

Figure 5-9. Smart phone functions realized by Proximity sensing

Radar-based gesture recognitionA radar-based gesture detection is currently being developed at Google under a name, project Soli. A tiny chip smaller than a dime (Figure 5-11) can detect subtle gestures such as sliding on an index finger with a thumb, tapping two fingertips together (Figure 5-10), or crossing two fingers each other (“Project Soli”, 2016). According to an article (Pierce, 2016), the technology itself is way ahead of the implementation, and looking forward to the commercialization only in few years.

The possibility of micro-gesture detection from this project has a big implication to the feasibility of the design concept. If this technology is embedded in the smartphone, finger motion will be detected more effective. This will enable some functions (e.g. Activate the camera, Launch the recorder) that require the recognition of more complicated micro-gestures. It is needless to say that other

FINAL DESIGN

97


functions that require simpler gesture recognition benefits from the same technology.

5.4.2. CAMERA BASED GESTURE RECOGNITION

Image processingThe front camera embedded in a smartphone can be used for the gesture recognition. This can be realized by using image processing technology. Various image processing algorithms, such as color tracking, motion tracking, template matching, blob tracking, and multiple cues integrating (Wu & Huang, 1999) enable the detection of various gestures.

Switch control feature in iPhone allows the detection of head movements, either to the left or right. Realized through a camera, it helped people with limited mobility to be able to use iPhone (Russell, 2016). Various applications (“CamMe - hands free camera app for iOS”, 2016; “Snapi – eyesight”, 2016; “G-Selfie”, 2016) offer functionality to take a picture by making a fist, making a V-sign, opening and closing the mouth, etc. without touching the screen or pressing a button.

Features and gestures such as open hand, closed hand, thumbs-up, V-sign, face, wink, grab, drop, swipe, by using the video information gathered from the embedded camera of a mobile device (“Touchless A3D – Crunchfish”, 2016).

5.4.3. PEER TO PEER COMMUNICATION

Infrared technologyInfrared technology traditionally used for remote controls proves the feasibility of point-and-connect. IR technology is one of the mature technologies that enables point-and-connect which has been commercially available since the 1980s (“Remote Background - Zenith Electronics”, 2016). One of the well-known pitfalls of IR remote control is the limitation in operating angle. This turns into an advance for this concept.

Through this technology, connecting with a device by pointing (point-and-connect), and which device was connected (target distinction) are considered feasible.

Many Smartphones already offer similar functionality by the embedded IR blaster. An IR Blaster emulates the functionality of remote for any device, making a smartphone as a universal remote control for various home appliances (Verma, 2014). For example, this can allow the control of Smart TVs without a dedicated remote control device.

5.4.4. CONCLUSION

It is concluded that the design concept is technologically feasible with the combination of technologies that are already available on the market.

The functions such as: ‘Wake up the device’, ‘make a call’, ‘answer a call’, ‘Snooze a call’, ‘Reject a call’, ‘Snooze the alarm’, and ‘Turn off the alarm’ are proven to be feasible from the existing examples. More complicated functions that require finger detection (e.g. Launch the messenger, Active the camera, Launch the recorder) are considered technologically feasible as well by using the combination of different inputs from various sensor inputs. Additionally, when the future technology is looked into, the micro gesture recognition that is close to the commercialization is expected to increase the accuracy and efficiency of the design. For the ‘Launch the music app’ function, no particular technology was considered to be required to detect an earphone being plugged. For the ‘Start remote control’ function, rather simple technology such as Infrared sensor can make the feature feasible.

There are various ways to implement the design technologically.

The efficiency of gesture recognition in terms of accuracy, speed, and diversity in gesture recognition will be crucial in choosing the appropriate technology. The battery consumption and the cost of implementation will influence this as well.

Figure 5-10. Project Soli, Radar based gesture recognition

Figure 5-11. A sensor tiny as a dime used for the radar-based gesture recognition

EVALUATIONEvaluation study assesses the final design with the users. The study setup, results and analysis are covered in this chapter.

6

CHAPTER 6.

100

Related StudiesVarious studies were looked into in advance of the final evaluation study. Many studies (Lee, Chae, Kim, Lim, & Lee, 2013; Vatavu, 2012; Kühnel et al., 2011) took an exploratory approach to ask the participants to come up with a gesture for a certain function. Such user-defined gesture libraries were used to prove their credibility. Other studies (Wobbrock, Wilson & Li, 2007; Cao & Balakrishnan, 2005; Baudel & Beaudouin-Lafon, 1993) focused on the evaluation of pre-defined gestures concerning user and system performance. They discussed criteria such as the trajectory of gestures or the performance time to for the assessment. The studies involved more technologies in building a gesture recognition system for the assessment. One study evaluated a pre-defined set of gestures in different social contexts (Rico & Brewster, 2010). The participants were presented with gestures with videos, then encourage to act and evaluate the gestures regarding its usages in social contexts.

OBJECTIVESIn this study, the study aimed to assess the correctness of the gesture-function mapping, appropriateness of the feedback, relevance with Smart living. A technical prototype with a complicated gesture recognition system was not able to be built within a short amount of time, due to the complexity and the subtleness of the gestures. The gestures for both of the first and the second part of the study discussed four different topics; Performance, Gesture, Feedback, and Smartness.

The performance assessed the performance of the gestures. The gesture part included Intuitiveness, ease of use, awkwardness at home, and awkwardness in public. Intuitiveness considered mental easiness, whether a gesture is easy to understand. The ease of use focused on evaluating physical easiness, whether a gesture is physically easy to perform, regarding the movements and motor skills required. Different environments such as home and public and different conditions such as being alone and being with other people were considered when assessing awkwardness.

The feedback part assessed the haptic feedback realized by vibration, visual feedback realized by LED, and the mapping. The haptic and visual feedback regarded the general feeling of each. The mapping evaluated whether any feedback given is helpful to understand the interaction.

The smartness part assessed if the use of gestures feels ‘smart’. The criteria included: shortcut, general feeling, drag-and-drop, and expressiveness. Each of them evaluated whether the gestures serve as a shortcut, whether the drag-and-drop function feels smart, and how the responsiveness to speed feels, respectively. Rather than an assessment based on a numeric system, the research aimed to be a qualitative one with interviews.

METHODParticipantThree master students studying at TU Delft Industrial design engineering participated in this final evaluation study. They were not exposed to any other user research for this thesis project before. Their age was from 27-32, with three different nationalities which were Korea, America, and Mexico. Everyone owned a smartphone and actively using it on a daily basis, while not having so much experience with Smart living or Gesture control. Two participants were right-handed, and one participant was left-handed.

ProcedureThe evaluation study consisted of two parts. The first part focused on the evaluation of the smartphone gesture control library. The second part concentrated on the evaluation of the remote control gestures for home environments. An iPhone 4s was used in the study. A vibration module was attached to simulate haptic feedback (See Figure 6-4). For both of the study, Wizard-of-Oz method was used for the gesture recognition. The wizard could control the functions, visual and haptic feedback according to the gesture of the participants from one computer. It aimed to reduce the number and effort of the wizard. The system for the test was built using Processing (See Figure 6-3).The setup was carefully designed for the evaluation test through the reflections learned from a pilot study. The original setup for the pilot study, insights and reflections can be found in Appendix I. Pilot Study. Figure 6-1 illustrates an overview of the evaluation study setup, and Figure 6-6 shows one scene from one of the user tests.

In the first part, a video clip demonstrating twelve different functions of the smartphone gesture control library was presented. A participant was asked to trigger a certain function using the gesture library. The functions or the tasks were given in a random order afterward. The wizard controlled the feedback accordingly. The performance was noted to assess the intuitiveness of the gesture library. A short interview session followed after the first part.

The second part of the study, a participant was asked to control a few home appliances using a smartphone as a gesture controller. Similar to the first study, a demonstration video (See Figure 6-5) was presented. A living room with the home appliances – a vertical blind, horizontal roller blind, speaker, thermostat, and three ceiling lamps – was projected on a wall to simulate (See Figure 6-2). The main reason to use a projection instead of a real setting was to minimize the number of the wizard by building one integrated prototype. Each of the appliances had a dot which served as an LED. After the second test, another interview session was held for extensive feedback. Detailed interview guideline and questions can be found in Appendix J. Evaluation Study Guideline.

6.1.SETUP

EVALUATION

101


Figure 6-1. Evaluation study setup

Figure 6-2. The projection simulating a living room with appliances

Figure 6-4. Vibration module attached to the smartphone

Figure 6-3. Processing: Prototyping tool used for the simulation

Figure 6-5. Participant controlling the roller blind in the user testFigure 6-5. Participant watching the demonstration video

CHAPTER 6.

102

6.2.RESULTS AND ANALYSIS

RESULTSPart #1 Smartphone gesture controlTable 6-1. Shows the performance of the smartphone gesture control. Most of the gestures were understood, remembered and performed well by the participants. Wake up, play music, snooze and turn off functions were the exceptions. Two participants failed to perform the Wake up the device gesture. Two participants failed to perform play music gesture. One of the participants got confused and swapped snooze and turn off gestures, or performed hesitantly, for both the call and alarm.

Participant 1 Participant 2 Participant 3

Wake up X O X

Make a call O O O

Answer a call O O O

Snooze a call O O ∆

Reject a call O O ∆

Launch the messenger O O O

Activate the camera O O O

Snooze the alarm O O ∆

Turn off the alarm O O ∆

Launch the recorder O O O

Launch the music app X X O

Start Remote control O O O

Table 6-1. Smartphone gesture control performance results

O perfect performance ∆ Performance with errors X Poor or no performance

EVALUATION

103


Gesture controlWhen it comes to gesture control, many gestures were evaluated to be intuitive, except for the snooze/reject a call, and snooze/turn off the alarm functions. One participant who got confused snoozing and the other functions mentioned that those functions are confusing if not carefully thought. However, he also said that it can be quickly and easily learned from the first time use, and the current mapping makes more sense than he performed when the physical efforts are considered. Two other participants evaluated all the gestures to be very intuitive. One mentioned that the gestures were very intuitive, as most of the gestures were already used, being a part of the smartphone usage pattern.

The ease of use was evaluated to be very good for the most of the gestures. Only send a message and activate the camera gestures got remarks from the participants that using two hands are always less easy than using one hand, but it is still not at a challenging level at all.

The awkwardness at home was evaluated to be very good for all the gestures. The awkwardness in public yielded similar results, only except for the Make a call gesture. For the Make a call gesture, one participant mentioned the situation of using a nickname as a contact name and having to say that in public could be awkward. The

subtleness of the gestures, which came for two reasons; gestures being a part of the actions that are already being done, and the movements being small and subtle, were mentioned as the reasons for the good remarks.

FeedbackHaptic feedback was understood correctly and evaluated good by everyone.

SmartnessAll of the gestures except for the Activate the camera function was considered as a shortcut to execute a function. One participant mentioned the Activate the camera function as an exception. It was said that the camera function is already made to be very easy to access by swiping up from the lock screen in iPhones, which almost costs no effort.

Make a call, answer a call gestures were seen to evoke some insecurity by accidentally executing the function. Launch the music app and Start remote control function were evaluated mediocre, because of the preference to play music on speaker and visibility of the remote control gesture caused by approaching gesture, respectively.

Gesture control Feedback Smartness

Intuitiveness Ease of use HomeAwkwardness

PublicAwkwardness

Haptic(Vibration) Shortcut General

Feeling

Wake up O O O O O O O

Make a call O O O O O O ∆

Answer a call O O O O O O ∆

Snooze a call ∆ O O O O O O

Reject a call ∆ O O O O O O

Launch the messenger O ∆ O O O O O

Activate the camera O X O O O ∆ O

Snooze the alarm ∆ O O O O O O

Turn off the alarm ∆ O O O O O O

Launch the recorder O O O O O O O

Launch the music O O O O O O ∆

Start remote control O O ∆ O O O ∆

O Good with no complaints ∆ Medium with some complaints X Poor with many complaints or crucial complaint

Table 6-2. The result of the smartphone gesture control performance

CHAPTER 6.

104

Part #2 Remote control gestures for home environmentsIn the second study to evaluate the remote control gestures for home environments, all the participants were able to perform the gesture correctly. The participants mentioned that the use of universal gestures for multiple devices made it very easy to remember the gestures.

Gesture controlThe cases of lamp(s), speaker and thermostat were evaluated to be less intuitive than the roller blind, vertical blind. One participant mentioned that he would prefer a circular movement rather than a linear movement, using a knob metaphor. The thermostat was considered even less intuitive, as the change in temperature does not show any instant effect. The ease of use was evaluated to be mediocre. Pointing at an appliance already made the participants stretch their arm, the push forward action made it more extreme, raising fatigue issue during the adjustment (rolling up and down, or opening and closing) tasks.

FeedbackIn the second study, the consistent haptic feedback created some annoyance. A Pulse type vibration was recommended over the

constant vibration. The visual feedback was considered necessary for the connection confirmation, however only as an additional assurance. Most of the time the participants said that they were looking the changes in the real appliances, such as the blinds opening up, or lamps brightening up, as it is easier to see the changes in the actual appliances that the small LEDs. However, the thermostat was an exception. As the thermostat was located far in the scene and the change in the temperature cannot be sensed instantly, the participants felt the lack of visual feedback.

SmartnessDuplication gesture was not considered as a shortcut at all by one participant. He mentioned one particular case; when turning off all the lights at once, pressing physical switches all at once would take a second while the gesture controlling would take more time.

The participants evaluated that the gestures generally feel smart and contributing to the smart living, only except for the thermostat which lacked feedback crucially.

The expressiveness, the responsiveness to speed, was not considered smart. First of all, it was thought unnecessary. One participant said that he does not get the relation between the distance, angle, and speed, thus it would create more confusion.

O Perfect performance ∆ Performance with errors X Poor or no performance

Roller blind Vertical blind Lamp Speaker Thermostat Duplication

Participant #1 O O O O O O



Table 6-3. Remote control gesture performance results

EVALUATION

O Good with no complaints ∆ Medium with some complaints X Poor with many complaints or crucial complaint

Roller blind Vertical blind Lamp Speaker Thermostat Duplication

GestureControl

Intuitiveness O O ∆ ∆ ∆ O

Ease of use ∆ ∆ ∆ ∆ ∆ ∆

Social awkwardness O O O O O O

Feedback

Haptic (vibration) ∆ ∆ ∆ ∆ ∆ ∆

Visual (LED) O O O O X O

Mapping O O O O O O

Smartness

Shortcut O O O O O X

General feeling O O O O ∆ ∆

Expressiveness X X X X X X

Table 6-4. Remote control gesture performance results

105


ANALYSIS AND DISCUSSIONThe analysis focused on the few negative feedbacks for the reflection. The final study raised few issues to consider which are discussed below.

Automation evoking insecurity for the calling functionsGesture control may evoke insecurity due to its automation property, but it is not considered crucial. Gesture control offers a shortcut for executing a function. This may make the users feel less secure regarding calling functions (e.g. Make a call, Answer a call, Snooze a call, Turn off a call), as users can make human errors when performing gestures and calling is considered an irreversible event. However, this is not considered as a crucial problem of the design, as it is dependent on the user. The user can learn the system better, and if he/she keep makes errors, the design provides an option to turn off the function.

Confusion between Snooze and Turn off functionAs figured out in the previous researches during the iterations, one participant got confused about these functions again. Due to the same reason, this is again considered as functions that user can get quickly accustomed to.

Lack of functional/visual feedbacks in the case of thermostatA traditional way of controlling through application or Sound feedback is considered as a solution for the better user experience.

Thermostat Gesture control was evaluated less intuitive and less easy to use. First of all, the change in temperature cannot be felt immediately. Secondly, the simple and visible feedback was not delivered successfully due to the long distance between the user &

the device, and the small size of the device interface, especially at such distance. This seems to have a significant influence although haptic feedback is still provided.

Fatigue issue from approaching and moving up/downFatigue issue due to the approaching gesture suggests the approaching distance should be minimal. Moving up and down the arm raised fatigue issues to some of the participants, especially when interacting with a big object located close by (e.g. roller blinds).

Duplication gesture not thought as a shortcutThis seems to be dependent the personal preference, whether which of the tasks between walking toward the physical switch and long duration of interaction are more effort demanding to the user. As far as the users are given with whichever the options to choose as the design does not substitute the original physical interface it was seen as a trivial issue.

Implications for the mapping of the speedThe Wizard-of-Oz method simulated the speed of the interaction based on the wizard’s approximation of the speed. This confused some of the participants and hindered them in understanding the exact mapping of the speed. When regarding the change of the location (e.g. open/close roller blinds), three different mappings can be thought out; absolute mapping to move according to the real distance of the smartphone movement (Figure 6-6), relative mapping based on the angle displacement of the smartphone movement (Figure 6-7), and constant mapping that the target always moved at a predefined speed (Figure 6-8). It is thought that providing the constant speed for the remote control might be the more technologically feasible and less complicated solution to the users.

Figure 6-6. Absolute distance mapping: Target movement is mapped with the phone’s distance displacement

Figure 6-7. Relative distance mapping: Target movement is mapped with the phone’s angular displacement

Figure 6-8. Constant speed mapping: Target moves up on a constant speed when the phone is held up

A

A

B

B

A’ A

A” A

Figure 6-6. Absolute distance mapping: Target Figure 6-7. Relative distance mapping: Target Figure 6-8. Constant speed mapping: Target moves

CONCLUSIONThis chapter includes the conclusion from the evaluation study, and this links to the assessment of the final design in respect of the original assignment. The recommendations are suggested for the future direction in the field of Gesture control.

7

CHAPTER 7.

108

7.1.CONCLUSION

Smart living offers design opportunities in various domains utilizing various technologies. This thesis aimed to introduce gesture control in the smart living context and took an exploratory approach seeking the way to benefit Smart living through gesture control.

The research found out that the users expect automation and intelligence when it comes to smart living at home environment. Intelligence was not feasible as the gesture control requires the user input, thus the combination of gesture control and automation was explored. The ideation started from introducing gesture control to a smartphone, which is already a familiar device that are considered smart and closely related to our daily living.

The final design provides a set of smartphone gestures that controls home environments including the control of the smartphone itself. Through an iterative process, the concept was polished based on the assessment on smartness, which regarded intuitiveness, ease of use, and natural (as opposed to social awkwardness). The core strategies of the design are being a part of the original actions and resemble general custom and personal rituals.

The final evaluation study with an interactive prototype demonstrated the design contributes the smart living by providing convenience and comfort through a gesture control that are intuitive, physical and mentally less demanding while not evoking any social awkwardness by performing gestures in a given context.

In conclusion, this master thesis, “Smartphone controlled home environments” achieved an interaction design that shapes user’s life smarter, by providing intuitive and effortless gesture control realized through a smartphone that is applicable to both of the smartphone and home environments.

CONCLUSION

109


7.2.RECOMMENDATIONS

Lastly, few recommendations are proposed.

The first one is a further research on the haptic feedback. Some revisions are recommended for the design of the haptic feedback especially regarding the remote control function. The interaction duration of the remote control is longer than the smartphone control, and the constant vibration evoked annoyance. The pulse type vibration is proposed while varying the intensity depending on the input (e.g. pointing, approaching, on/off, adjustment). However, vibration has various properties to consider such as intensive, duration, and location, an individual research on such haptic feedback is recommended to support this thinking.

The second one is more exploration on the expressiveness. The expressiveness in this thesis means the interaction speed especially for the remote control function, which allows the target object to be controlled based on the gesture performance speed. During the final study, the speed of the interaction was explored only on an exploration level. The final study assessed the basic idea of it through interviews with the participants, without any extensive evaluations without any precise simulation. Setting a constant speed for the remote control seems the most feasible and easy solution based on the findings. However, it still leaves a space for further study. More explorations on the speed-interaction mapping will lead to better user experience.

Lastly, when it comes to the implementation of the final design, technical details will have to be figured out, in order not to hinder the user experience due to technological constraints.

REFERENCES 8

112

REFERENCES

ACTIVE AND ASSISTED LIVING PROGRAMME. (2016). AAL - ACTIVE AND ASSISTED LIVING PROGRAMME. Retrieved 13 December 2015, from http://www.aal-europe.eu/

Aldrich, F. K. (2003). Smart homes: past, present and future. In Inside the smart home (pp. 17-39). Springer London

iTunes - iTunes Charts - Apple. (2015). Apple. Retrieved 20 November 2015, from http://www.apple.com/itunes/charts/

Arcus Motion. (2016). Arcus Motion Analyzer - activity tracker and gesture control device. Arcusmotion.com. Retrieved 28 September 2015, from http://arcusmotion.com/

Auld, N. (2012). A SIRIES OF UNFORTUNATE EVENTS: Users’ Emotional Responses during the First Month of Siri Use. blinkux. Retrieved 4 September 2015, from https://blinkux.com/_downloads/Blink_Siri_White_Paper.pdf

Baby Sign Language. (2016). BabySignLanguage.com. Retrieved 21 February 2016, from http://www.babysignlanguage.com/

Baby Sign Language. (2016). Parents.com. Retrieved 21 September 2015, from http://www.parents.com/baby/development/sign-language/#page=2

Ballmer, S. (2016). CES 2010: A Transforming Trend - The Natural User Interface. The Huffington Post. Retrieved 25 September 2015, from http://www.huffingtonpost.com/steve-ballmer/ces-2010-a-transforming-t_b_416598.html

Baudel, T., & Beaudouin-Lafon, M. (1993). Charade: remote control of objects using free-hand gestures. Communications of the ACM, 36(7), 28-35.

Baxter, L. K. (2000). Capacitive sensors. Ann Arbor, 1001, 48109.

Billinghurst, M., & Buxton, B. (2011). Gesture based interaction. Haptic input, 24.

BoeBoer - Phone & Tablet’s Manuals Review & Download. (2015). Boeboer.com. Retrieved 25 November 2015, from http://www.boeboer.com/

Boeijen, A., Daalhuizen, J., Zijlstra, J., & Schoor, R. (2013) Delft design guide. Amsterdam: BIS Publishers.

Bolt, R. A. (1980). “Put-that-there”: Voice and gesture at the graphics interface (Vol. 14, No. 3, pp. 262-270). ACM.

Borenstein, G. (2012). Making things see. Farnham: O’Reilly

Briant, M. (2004). Baby sign language basics. Carlsbad, Calif.: Hay House.

CamMe - hands free camera app for iOS. (2016). Pointgrab.com. Retrieved 26 February 2016, from http://www.pointgrab.com/camme/

Cao, X., & Balakrishnan, R. (2005, May). Evaluation of an on-line adaptive gesture interface with command prediction. In Proceedings of Graphics Interface 2005 (pp. 187-194). Canadian Human-Computer Communications Society.

Capone, N. C., & McGregor, K. K. (2004). Gesture Development: A Review for Clinical and Research Practices. Journal of Speech, Language, and Hearing Research, 47(1), 173-186.).

Child using Gestures to communicate. (2016). Youtube.com. Retrieved 21 February 2016, from https://www.youtube.com/watch?v=uN_-R_4fKWY

Clover, J. (2015). Inside iOS 9: A More Intelligent OS With Proactive Suggestions and Siri Improvements. MacRumors. Retrieved 28 September 2015, from http://www.macrumors.com/2015/09/16/ios-9-proactive-suggestions/

Costello, S. (2016). The 6 Sensors in an iPhone. About.com Tech. Retrieved 26 February 2016, from http://ipod.about.com/od/ipodiphonehardwareterms/qt/iphone-sensors.htm

Desmet, P., & Hekkert, P. (2007). Framework of product experience. International journal of design, 1 (1) 2007.

Exclusive: Proximity Sensing in Mobile Phones. (2014). Wireless Design and Development. Retrieved 15 January 2016, from http://www.wirelessdesignmag.com/article/2014/01/exclusive-proximity-sensing-mobile-phones

Eye tracking. Wikipedia. Retrieved 16 September 2015, from https://en.wikipedia.org/wiki/Eye_tracking

Feature phone. Wikipedia. Retrieved 16 September 2014, from https://en.wikipedia.org/wiki/Feature_phone

FIN Robotics. (2016). Finrobotics.com. Retrieved 28 September 2015, from http://www.finrobotics.com/

Frommer, D. (2015). These are the 25 most popular mobile apps in America. Quartz. Retrieved 20 November 2015, from http://qz.com/481245/these-are-the-25-most-popular-2015-mobile-apps-in-america/

G-Selfie (2016). Macrongesture.com. Retrieved 26 February 2016, from http://www.macrongesture.com/products_04.html

Gest. (2016). Gest.co. Retrieved 28 February 2016, from https://gest.co/

Gesture Remote. (2015). Gesture-remote.com. Retrieved 28 September 2015, from http://www.gesture-remote.com/

Gesture. Oxforddictionaries. Retrieved 8 July 2015, from http://www.oxforddictionaries.com/definition/english/gesture

Goldstein, E. (2010). Encyclopedia of perception. Thousand Oaks, Calif.: Sage Publications.

Greenwald, W. (2015). Amazon Echo. PCMAG. Retrieved 15 November 2015, from http://www.pcmag.com/article2/0,2817,2476678,00.asp

Grudin, J. (2005): Three faces of Human-Computer Interaction. Annals of the History of Computing, vol. 2 no 4, 2-18.

Harper, R. (2003). Inside the smart home. London: Springer.

Harper, R. (Ed.). (2006). Inside the smart home. Springer Science & Business Media.

Home automation. Wikipedia. Retrieved 15 September 2015, from https://en.wikipedia.org/wiki/Home_automation

Honan, M. (2015). Siri Is Apple’s Broken Promise. Gizmodo. Retrieved 20 September 2015, from http://gizmodo.com/5864293/siri-is-apples-broken-promise

Hornecker, E., & Buur, J. (2006). Getting a Grip on Tangible Interaction: A Framework on Physical Space and Social Interaction. Proceedings Of The SIGCHI Conference On Human Factors In Computing Systems, 437-446. doi:10.1145/1124772.1124838

How A Simple Apple Feature Called Switch Control Is Changing Lives. (2016). Buzzfeed.com. Retrieved 15 January 2016, from http://www.buzzfeed.com/charliewarzel/how-a-simple-apple-feature-called-switch-control-is-changing#.qlY476EGj5

How does “Air View” feature work?. (2015). Quora.com. Retrieved 25 November 2015, from https://www.quora.com/Samsung-Galaxy-S4/How-does-Air-View-feature-work

Income, S. (2016). Newswire | Survey: New U.S. Smartphone Growth by Age and Incom | Nielsen . Nielsen.com. Retrieved 15 January 2016, from http://www.nielsen.com/us/en/insights/news/2012/survey-new-u-s-smartphone-growth-by-age-and-income.html

Intugine Technologies. (2015). Intugine.com. Retrieved 28 September 2015, from http://www.intugine.com/

iOS: Switch Control helps you navigate your iOS device - Apple Support. (2015). Support.apple.com. Retrieved 25 November 2015, from https://support.apple.com/en-us/HT201370

Kendon, A. (1980). Gesticulation and speech: Two aspects of the process of utterance. The relationship of verbal and nonverbal communication, 25, 207-227.

Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University Press.

Klemmer, S. R., Hartmann, B., & Takayama, L. (2006, June). How bodies matter: five themes for interaction design. In Proceedings of the 6th conference on Designing Interactive systems (pp. 140-149). ACM.

Koster-j, J. (2006). Applying gestures in domestic appliances.

REFERENCES

113


Krug, S. (2013). Don’t make me think!. Berkley: New Riders.).

Kühnel, C., Westermann, T., Hemmert, F., Kratz, S., Müller, A., & Möller, S. (2011). I’m home: Defining and evaluating a gesture set for smart-home control. International Journal of Human-Computer Studies, 69(11), 693-704.

Leap Motion. (2015). Leapmotion.com. Retrieved 18 September 2015, from https://www.leapmotion.com/

Lee, S. S., Chae, J., Kim, H., Lim, Y. K., & Lee, K. P. (2013, September). Towards more natural digital content manipulation via user freehand gestural interaction in a living room. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing (pp. 617-626). ACM.

Mann, S. (2002). Intelligent image processing. New York: IEEE.

Mayhew, D. (2008). User Experience Design: The Evolution of a Multi-Disciplinary Approach. The Journal Of Usability Studies, 3(3), 99-102.

McNeill, D. (2000). Language and gesture (Vol. 2). Cambridge University Press.

Meese, J. (2015). How to Activate Air Gestures on the Samsung Galaxy S4. Retrieved 25 November 2015, from http://johnmeese.me/galaxy-s4-air-guestures/

Mritfit.com,. (2015). Use your mobile with Air Gesture. Retrieved 25 November 2015, from http://www.mritfit.com/use-your-mobile-with-air-gesture/

Myers, B., Hudson, S., & Pausch, R. (2000). Past, present, and future of user interface software tools. ACM Transactions On Computer-Human Interaction, 7(1), 3-28. doi:10.1145/344949.344959

Myo Gesture Control Armband. (2015). Myo.com. Retrieved 18 September 2015, from https://www.myo.com/

Natural user interface. Wikipedia. Retrieved 16 September 2015, from https://en.wikipedia.org/wiki/Natural_user_interface

Natural user interface. Wikipedia. Retrieved 27 February 2016, from https://en.wikipedia.org/wiki/Natural_user_interface

Nest. (2015). Nest. Retrieved 13 December 2015, from https://nest.com/

nod. (2016). Home - nod. Retrieved 28 September 2016, from https://nod.com/

Norman, D. A. (2013). The design of everyday things: Revised and expanded edition. Basic books.

Özçalışkan, Ş., & Goldin-Meadow, S. (2005). Gesture is at the cutting edge of early language development. Cognition, 96(3), B101-B113.

Pierce, D. (2016). Can Google Fix Gesture Tech With Tiny, All-Knowing Sensors?. WIRED. Retrieved 27 February 2016, from http://www.wired.com/2015/05/google-atap-project-soli-gesture-technology/

Project Soli (2016). Google.com. Retrieved 15 January 2016, from https://www.google.com/atap/project-soli/

Proximity sensor on Galaxy S4 (Air Gestures) (2015). Stackoverflow.com. Retrieved 25 November 2015, from http://stackoverflow.com/questions/16649830/proximity-sensor-on-galaxy-s4-air-gestures

Remote Background - Zenith Electronics. (2015). Zenith Electronics. Retrieved 15 September 2015, from http://www.zenith.com/remote-background/

Revolutionary User Interfaces. (2015).Timeline.knightlab.com . Retrieved 16 September 2015, from https://timeline.knightlab.com/examples/user-interface/

Rico, J., & Brewster, S. (2010, April). Usable gestures for mobile interfaces: evaluating social acceptability. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 887-896). ACM.

Roozenburg, N. F., & Eekels, J. (1995). Product design: fundamentals and methods (Vol. 2). Chichester: Wiley.

Rouse, M. (2005). What is user interface (UI)?. SearchSOA. Retrieved 16 September 2015, from http://searchsoa.techtarget.com/definition/user-interface

Russell, K. (2016). How To Control Your iPhone By Tilting Your Head. Business

Insider. Retrieved 15 January 2016, from http://www.businessinsider.com/control-your-iphone-with-head-gestures-2014-3?IR=T

Sacco, A. (2015). The 16 Most Popular Mobile Apps in the World. CIO. Retrieved 20 November 2015, from http://www.cio.com/article/2370318/mobile-apps/the-16-most-popular-mobile-apps-in-the-world.html

Samsung GALAXY S4 - Life Task. (2015). Samsung.com. Retrieved 25 November 2015, from http://www.samsung.com/global/microsite/galaxys4/lifetask.html#page=airview

Siri Archives - Intelligent Voice Blog - Speech Technology. (2013). Intelligent Voice Blog - Speech Technology. Retrieved 18 September 2015, from http://www.intelligentvoice.com/blog/tag/siri/

Smart Home. Intelligent Living.. (2015). SmartThings. Retrieved 13 December 2015, from https://www.smartthings.com/

Snapi - eyeSight. (2016). eyeSight. Retrieved 26 February 2016, from http://eyesight-tech.com/snapi/

Solaimani, S., Keijzer-Broers, W., & Bouwman, H. (2013). What we do–and don’t–know about the Smart Home: an analysis of the Smart Home literature. Indoor and Built Environment, 1420326X13516350

T, N. (2014). Did you know how many different kinds of sensors go inside a smartphone?. Phone Arena. Retrieved 15 January 2016, from http://www.phonearena.com/news/Did-you-know-how-many-different-kinds-of-sensors-go-inside-a-smartphone_id57885

Tecla,. (2016). What is Switch Control mode in Apple’s iOS?. Retrieved 15 January 2016, from http://gettecla.com/blogs/news/15538916-what-is-switch-control-mode-in-apples-ios

Thal, D., & Bates, E. (1988). Language and gesture in late talkers. Journal of Speech, Language, and Hearing Research, 31(1), 115-123.

The Best Computer Interfaces: Past, Present, and Future | MIT Technology Review. (2009). MIT Technology Review. Retrieved 14 September 2015, from http://www.technologyreview.com/news/412880/the-best-computer-interfaces-past-present-and-future/

Touchless A3D - Crunchfish. (2016). Crunchfish. Retrieved 26 February 2016, from http://crunchfish.com/touchless/

Tweedie, S. (2015). The 10 Most Popular Apps Of 2014. Business Insider. Retrieved 20 November 2015, from http://uk.businessinsider.com/the-most-popular-apps-of-2014-according-to-nielsen-2014-12?op=1

Vatavu, R. D. (2012, July). User-defined gestures for free-hand TV control. In Proceedings of the 10th European conference on Interactive tv and video (pp. 45-48). ACM.

Verma, A. (2014). List of Android Smartphones with IR Blaster. Akshatblog.com. Retrieved 27 February 2016, from http://www.akshatblog.com/list-of-android-smartphones-with-ir-blaster/

Wobbrock, J. O., Wilson, A. D., & Li, Y. (2007, October). Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the 20th annual ACM symposium on User interface software and tech-nology (pp. 159-168). ACM.

Wu, Y., & Huang, T. S. (1999). Vision-based gesture recognition: A review. In Gesture-based communication in human-computer interaction (pp. 103-115). Springer Berlin Heidelberg.

Xbox | Games and Entertainment on All Your Devices. (2016). Xbox.com. Retrieved 27 February 2016, from http://www.xbox.com/en-US/

Yan, X., & Aimaiti, N. (2011). Gesture-based interaction and implication for the future. Retrieved from http://umu.diva-portal.org/smash/get/diva2:437916/FULLTEXT01.pdf

Yarow, J. (2016). Ex-Apple Insider: ‘People Are Embarrassed By Siri’. Business Insider. Retrieved 18 September 2015, from http://www.businessinsider.com/ex-apple-insider-people-are-embarrassed-by-siri-2012-5?IR=T

APPENDICES9

SENSITIZING MATERIAL

USER INTERVIEW GUIDE

USER INTERVIEW SUMMARIZED RESULTS

IDEATION 1: ITERATIVE PROCESS

USER TYPOLOGY

IDEA SKETCHES

INTERACTION PROTOTYPING

PRELIMINARY GESTURE EVALUATION STUDY

PILOT STUDY

EVALUATION STUDY GUIDELINE

A.

B.

C.

D.

E.

F.

G.

H.

I.

J.

116

A.SENSITIZING MATERIAL

APPENDIX A. SENSITIZING MATERIAL

117


118


119


120


121


122


123


124


125


126

Objectives The objective of this first field research was to understand current situations, and to identify user needs and opportunities for gesture control to be introduce at home as a new controlling method. The research on the current contexts meant understanding lifestyles, which included their living arrangement, household tasks or home activities, and appliances used at home performing aforementioned tasks and activities. The interviewees were encouraged to share their interaction with the products thereafter, about how they would control each of the products, and then reflect their experience. Through this process, knowledge about user attitude toward Smart Living and new technology in general, the problems of the current interfaces, user needs and potentially beneficial situations of using gesture control at home environment were gained. Research topic

1. Lifestyles to understand the current lifestyles of people in terms of housing, co-residents, activities taking place, and appliances being used.

2. Beneficial scenarios using gesture control to learn situations that gesture control can be beneficial

3. Gesture control -

4. User attitudes toward smart living technology to hear opinions toward adopting smart living technology in life

Research questions Research questions can be described as followings.

1. User profile

The interviewees were considered as ‘potential users’. However, since a potential target user group had not been defined yet, it was important for the interviewees to represent broad spectrum of the users, to gain diverse insights regarding people within various living situation. Through the user research, a main target group to put focus on was expected to be defined in the latter stage of the project, and user profile information collected from this research were to be utilized to determine the target user group.

a. What kind of people should be the main target user of gesture control?

2. Lifestyles How is the current lifestyles of people?

a. What kind of housing do users live?

B.USER INTERVIEW GUIDE

APPENDIX B. USER INTERVIEW GUIDE

127


b. Who are the co-residents of the users? c. What kind of activities are done at home? d. What kind of appliances are being used? e. How do people control appliances, and do they have any problem with specific

controlling methods? 3. Beneficial scenarios using gesture control

What are the situations that gesture control can be beneficial? a. Can the gesture control contribute to overcome short-handed situations? b. Can the gesture control contribute to overcome distance problems? c. Can the gesture control contribute to reduce controlling time? d. Can the gesture control bring more convenience? e. Can the gesture control create more joyful experiences?

4. Gesture control experience (If already applied or being used) how are the user experience(acceptance) toward gesture control technology applied to home?

a. Are the users already have any gesture control related devices at home? b. If they have, what are their opinions toward the performances and user

experience? 5. User attitudes toward smart living technology

What are the user’s opinions toward Smart living, in terms of bringing new technology to life?

a. Are the users familiar with ‘Smart living’? b. How do they imagine ‘Smart living’? c. What kind of technologies are expected to come? d. What do the users think about those technologies being introduced at home?

Participants In total, 10 people were recruited for this research. For the diversity of the outcome, criteria such as age, gender, nationality , living arrangement, and technology adaptness were considered when selecting the participants. Methods Interviews with each participants took place at each of their residences. Firstly, participants were asked to give information about their housing, living arrangement, or coresidents of them. Afterwards, they were asked to explain further, focusing on each room of the house, detailing geography of each residence in the house and the activities being done. At the same time, the participant was also questioned to mention appliances in the rooms, and how and by whom they are utilized during aforementioned activities. After interactions between human and human, human and products are being verified, Audio/video recordings were made.

128

Questions 1. User profile

a. What kind of people should be the main target user of gesture control? i. Age ii. Gender iii. Nationality / Background

2. Lifestyles How are the current lifestyles of people?

a. What kind of housing do users live in? i. Could you tell me a bit more about your apartment? ii. Can you show me around the house and the rooms?

b. Who are the co-residents of the users? i. Whom are you living with?

c. What kind of appliances are being used? i. What kind of appliances, and (media) devices to you have in this space?

1. Could you explain what is it? 2. Could you describe when, or in what situation do you use it? 3. Who interacts with it, do you use it alone, or together with

someone else? 4. Could you describe how do you usually control it? 5. (If any) Can you describe some other ways to interact(control) with

the device? (ex. push buttons, remote control while sitting on a couch etc)

6. and what is your favorite method among all, and why is it so? 7. Could you describe any situations that makes you difficult/hard/not

able to use this device with the the method? 8. What was your intention, and what makes it hard? 9. What kind of actions do you make to solve the problem?

d. What kind of activities are done at home? i. What kind of activities do you or your coresidents do in this space? Can

you describe some main interactions in this space? 3. Beneficial scenarios using gesture control

What are the situations that gesture control can be beneficial? a. Can the gesture control contribute to overcome short-handed situations? b. Can the gesture control contribute to overcome distance problems? c. Can the gesture control contribute to reduce controlling time? d. Can the gesture control bring more convenience? e. Can the gesture control create more joyful experiences?

4. Gesture control experience (If already applied or being used) how are the user experience(acceptance) toward gesture control technology applied to home?

a. How much is the gesture control well-known to users?

APPENDIX B. USER INTERVIEW GUIDE

129


i. Do you know gesture control devices such as Kinect, Xbox, eye trackers, MYO band from thalmic lab, etc?

ii. Can you share some thoughts about gesture controlled devices, such as Kinect, Xbox, eye trackers, etc. if you have any experience with it?

iii. Could you describe any situation where you would like to apply gesture control?

b. Are the users already have any gesture control related devices at home? i. Do you have any gesture control device at home? or have you ever

purchased one? c. If they have, what are their opinions toward the performances and user

experience? i. (If using any gesture control device) What kind of benefits does this

device (supposed to) have? ii. Why did you get/purchase the device, and what was your expectations

then? iii. Does the device meet your expectation, and how do you feel while using

it, what grade you would give to its performance? iv. How actively are you using the device? v. Could you describe any situation that made your life so comfortable? vi. Could you describe any situation that it was not so convenient to use?

5. User attitudes toward Smart living technology What are the user’s opinions toward Smart living, in terms of bringing new technology to life?

a. Are the users familiar with ‘Smart living’? i. Do you have any smart/home automation devices? (If using any

smart/home automation devices) What kind of benefits does this device (supposed to) have?

ii. Does the device meet your expectation, and how do you feel while using it, what grade you would give to its performance?

iii. Could you describe any situations that makes you difficult/hard/not able to use this device with the the method?

iv. What was your intention, and what makes it hard? b. How do they imagine ‘Smart living’?

i. Can you share some thoughts about Smart living? c. What kind of technologies are expected to come?

i. What do you think about using smart products and its technology? d. What do the users think about those technologies being introduced at home?

i. Would you be willing to use it on your own or not, and could you explain some reasons behind?

6. People living with someone who has physical constraints a. What kind of (physical) constraints does the person has?

i. Could you explain what kind of constraints that he/she has in daily living? b. What opportunities are there to help the person better using gesture control?

i. Could you explain when they need some help with daily life tasks? ii. Could you describe any situations that you find it hard to give some help

to them? What needs to be collected?

Pictures of an interviewee Pictures of an apartment/house at each location Pictures/video of the controlling interactions/devices

i. Could you explain when they need some help with daily life tasks? ii. Could you describe any situations that you find it hard to give some help

to them? What needs to be collected?

Pictures of an interviewee Pictures of an apartment/house at each location Pictures/video of the controlling interactions/devices

130

2015

Respondent #01 Lexi

I pretty much like the idea of the smart living.I will give it a try, to see how it goes.“

Name Lexi

Age 24 years old

Primary needs: Being reminded, being healthy and comfortable Biggest Pain points: Forgetting daily tasks

“

Nationality Chinese

Living Arrangement 1 housemate / Apartment

Smart Living Experience Novice

Gesture Control Experience Intermediate

Technology Acceptance Intermediate

Household Appliances Basic

Gesture Control in a Holistic Home Environment

2015

LIFESTYLE

Respondent #01 Lexi


KITCHEN I cook, eat. I drink coffee, and listen to music. I listen to music with the little speaker. With my roommate, we cook together and socialize. ‣ Bluetooth Speaker

‣ I bring the speaker, and it is connected with bluetooth, either with my iPad or iPhone. To switch songs, I need to use iPad/iPhone. To adjust the volume, I can just use buttons on the speaker.

‣ It is difficult when my hands are dirty, otherwise, when my hands are clean, I don’t really have any problem.

‣ Stovetop ‣ If my roommate and I are cooking together,

it is not convenient that only one of us can use the best fire in the stove, the biggest one, in the front row. We don’t really use two fires in the back. and the other one in the front only has a very small fire.

‣ You press this, to make a spark. ‣ Then you need to turn the knob. ‣ I don’t like that you need two hands to turn

the fire on. ‣ It happens somethings that I forget the fact

that a fire is still on, when it is hidden under a pot.

‣ I don’t really bake. ‣ Kitchen hood

‣ - ‣ Coffee machine

‣ This is a special one, it has prepaid coffee pads.

‣ You switch it on, put a pad in, press it down and wait.

‣ I think it is well-made. ‣ What kind of coffee, how much coffee you

want, depends on what coffee pad you want.

‣ When there is no water, or not enough, I cannot have coffee. Nothing will happen if you press buttons then.

‣ Fridge ‣ -

‣ Microwave ‣ -

‣ Toaster ‣ I press the lever down. ‣ You can adjust the level. ‣ This had a really mechanic interface, and it

is already dirty, so I don’t really care. ‣ Ricecooker

‣ You just add water and rice, press it down, and wait.

‣ It is fine. ‣ This will be the first thing that I will do

before cooking, before getting my hands dirty.

‣ Watercooker ‣ -

‣ Washing machine ‣ This is quite easy.

‣ You select a function, then the temperature and the spinning speed.

‣ Actually, I don’t really use those two. ‣ I think it is kinda dirty here, that it is in the

kitchen close to the stove. we might drop something on it.

‣ Washing machine is a hygienic stuff, to be in the kitchen.

‣ Interface is fine. ‣ I bring laundries here, and before that I

always open the washing machine first. ‣ Light

‣ It is mechanical interface. ‣ It is easier for me to deal with a mechanical

interface, if I need to a simple task. for more complicated tasks, I prefer iPad/iphone.

‣ I like that iPad provides a lot of information for controlling.

‣ I think it also depends on what kind of stuff. ‣ iPad, speaker, are something that you want

to keep out from kitchen, to avoid making it greasy.

APPENDIX C.

C.USER INTERVIEW SUMMARIZED RESULTS

USER INTERVIEW SUMMARIZED RESULTS

131


2015

LIFESTYLE

Respondent #01 Lexi


MY BEDROOM I am sitting in front of laptop often, and I also here when I am home alone. I read a book. I dress up, wear a make-up… ‣ Ceiling lamp

‣ I really hate this ceiling lamp, since I don’t have a bulb for my bed lamp I always have to use the ceiling lamp.

‣ I usually read a book, or lie in bed, and I go to this sleeping phase, and then I have to wake myself up to turn off the light.

‣ My roommate switches it off if I forget to switch if off before going to bed.

‣ I have a problem with the wall switch, the switches are so close to each other, so I sometimes switch off the whole circuit instead of turning the light on.

‣ Studying Lamp ‣ When I am studying, working on my desk. ‣ It had a switch attached to it. ‣ If it mobile, when I need it I bring the whole

thing to my desk. ‣ Bluetooth Speaker

‣ I use to listen to music.

ROOMMATE’s ROOM (LIVINGROOM) Sometimes, when friends are coming, we eat here with the guests. We play music, drink, and socialize. We have a printer, and a speaker here. ‣ Speaker

‣ You just plug it in, switch on, and you are read to go.

‣ You can adjust things with these three knobs.

‣ Controlling can be done perfectly with one hand.

‣ But if I want to control, I control it from here but there(iPad)

‣ Controlling with iPad is hard when your hands are wet. I usually interact with it when I am cooking.

‣ I just don’t control it then. I just wish a better song will be played afterwards.

2015

Respondent #01 Lexi


BENEFICIAL SCENARIOS ‣ The lamp! I would love to use voice control. ‣ Maybe microwave. ‣ Because getting too close to the microwave is not

so healthy. I run away from the microwave once I start it.

‣ I think if people can gesture control everything, they will get so fat. they will never walk in their home.

‣ Music can be a good thing. ‣ If I am drinking with my friends, i always go to

iPad to control music, but if we can do it all together, with the light as well in a party…

GESTURE CONTROL EXPERIENCE ‣ Kinect/Xbox ‣ I have experience with Kinect/Xbox to play

games. ‣ It is actually hard, it was not really accurate. ‣ Before you play a game, you want to choose the

game by interface, It has this slide thing and it just went wrong. when I want to slide, it just clicked something, selecting another game instant.

‣ Playing games, while playing games, I was less conscious about the problem.

‣ Maybe controlling information requires more accuracy than just moving a figure.

‣ In the games, you have to choose a lot of things as well. tennis, surfing, a lot of stuffs, and then its very hard to choose things.

‣ You can move around, when you are moving in front of a screen.

‣ Even online coach. I think Nike is doing something with Xbox.

‣ Getting rid of a screen that only makes you sit and glare.

‣ Leapmotion ‣ It is kind of a thing that you are still sitting in a

desk. ‣ I don’t see a value of it yet. ‣ It is not really programmed well. ‣ A mouse is fine for me. ‣ I tried and its really bad. ‣ MYO band ‣ I only heard about it, but haven’t used it. ‣ But do you have to program by yourself?.. ‣ Hololens from Microsoft. ‣ It is something like google glass. ‣ You can use gesture control to control

information, virtual information, through the glass.

SMART LIVING EXPERIENCE ‣ I don’t have any smart devices here. ‣ I know Smart home from Bosch. ‣ What kind of foods are inside, when it is going to

expire. you can order things from the fridge. Voice input, i forgot the name…

‣ ‘add toilet paper in my shopping list’ ‣ It helps to manage your stuffs, and save cost in a

long run. ‣ I buy a lot of stuffs and it gets expired.. ‣ I pretty much like the idea of the smart living

ATTITUDE TOWARD SMART LIVING

‣ What I would think about future home, is for instance, you don’t need to bring keys anymore. and if you forgot, it won’t be a problem.

‣ It will remind you if you forgot anything, like my smart phone.

‣ It helps your house not to get burglars, or even prevent it.

‣ Technology ‣ If it is cheap enough, or if I am rich enough I will

(accept new technology at home). ‣ I will give it a try, to see how it goes.

Physical constraints ‣ The most difficult thing for my grandfather is to

bend down. ‣ He doesn’t do a lot of actives, since he is old. ‣ It is hard for him to take a bath. ‣ He sits in front of a TV, watching the TV for the

whole day. ‣ My mom interferes him to take a walk, switch of

the TV, forcing him to be more active. ‣ For him, controlling air conditioning, TV could be

controlled by gesture, but its not good for his health at all if he is given with that option.

132

2015

Respondent #02 Alexandra

Smart Living is to make my life obviously smarter for me, which means easier for me, things happening automatically. “

Name Alexandra

Age 26 years old

Primary needs: Making life easier, by automation Biggest Pain points: Trivial tasks ruining good moments

“

Nationality Danish

Living Arrangement Housemate / Apartment (1 housemate upstairs)


Gesture Control Experience Medium

Technology Acceptance Progressive



2015

LIFESTYLE



We share the bathroom, living room, and the kitchen. My housemate has her bed room upstairs.

KITCHEN ‣ We obviously cook, with the stove or oven. ‣ We eat in the kitchen, which is not typical for a

kitchen in Copenhagen, and it is quite nice. ‣ It is funny that my roommate got surprised by not

having a microwave. ‣ I think there is a low adoption rate in Denmark.

maybe only for popcorns, maybe considered a bit low quality.

‣ Milk foamer ‣ You can choose different settings to make

different things. ‣ It lights up when I put something in it. ‣ I don’t have to worry about it to get the milk

spoiled. When its done, it’s done. ‣ I like it to be away, till when it is done. ‣ Sometimes it beeps when it is done, but if I am

watching a really good part of the movie, it would be really nice if I would be able to re-start while away. because I don’t want to come over again and wait again.

‣ I know it is really a luxury problem. ‣ Toaster ‣ Instead of putting breads in the oven, which takes

a lot of time and energy, its fairly easy and fast. ‣ It does not take very long, and I have to be here

to put breads in, so I cannot really think of any luxury problems here.

‣ I mean its only about waiting, right. We don’t want to wait for long and we would like to know how long it is going to take.

‣ ‘30 seconds’, displayed like a microwave is nice. ‣ Stove/Oven ‣ The stove, you turn it on, and you can use it right

away. ‣ If I am cooking too much, I can get a bit chaotic.

everything needs to be cooked in different temperatures and time.

‣ It would be super cool to control like: this one, pasta, 50 g. one soft boiled egg. and if it could be done, it would be nice.

‣ But the oven, you need to wait to preheat. ‣ I want to know when it is done preheating. now it

only turns off its light. ‣ I usually do some other things like chopping the

vegetables. ‣ I think it is also something with my generation,

internet generation. ‣ It would be cool if we could just control the

temperature by doing this gesture. ‣ There are knobs to control, timer. ‣ The oven gets super hot. you have to be careful

not to touch the steel part accidentally, and when you open it.

‣ Dishwasher ‣ I don't really use my dishwasher. because I think it

is really not needed only for myself. its quicker to do it myself.

APPENDIX C. USER INTERVIEW SUMMARIZED RESULTS

133


2015

LIFESTYLE



LIVINGROOM ‣ TV

‣ Old-school remote, has so many buttons, and no idea what they are for.

‣ On/Off, Volume, and input to go into Apple TV. I never watch normal TV even though It is free.

‣ I can do everything with my laptop except for turning the TV on.

‣ Sometimes, when I am too tired and lazy afterwork, I wouldn’t take my laptop but just simply use the remote control.

‣ Controlling while eating at the couch, when you are covered up with a duvet, when we have girls night lying down here around the floor, it is difficult.

‣ After cooking for an hour, and ready to eat at the couch, and controlling TV with the laptop is hard.

‣ Remote control, has to be pointed at the right spot, if there is anything something blocking, out of battery, would be annoying.

‣ I used to have a lot more things here, but I moved it all now for the Apple TV.

‣ Lying on the ground, with a duvet, duvet gets annoying to use a remote control as well.

‣ At my parents house, with four children, with mattered on the floor, who’s gonna put the DVD on, being out of the coziness, comfortable state is interfered.

‣ Apple TV

‣ With Apple TV, I can use it with my Mac computer, stream it on the TV. It is so easy, so fast, you don’t need a cable. which I think as a very nice thing. no HDMI cable.

‣ Comfortably controlling from far away. ‣ It is so much nicer feeling to control with

the computer, than a remote control. ‣ You can do more with your laptop. and

much easier, because of the visual, UI. like GUI interface. click click and go back.

‣ I am very happy with it. ‣ I did not have TV for so many years, and I

got it half an year ago, it is a very different experience to watch series in a TV than a laptop. More cinema feeling.

‣ Controlling with Laptop, it feels bulky. very big remote control.

‣ It gets hot, to put on your sofa or your laps. ‣ iPad is definitely better as a remote, also for

its longer battery, but you cannot stream from iPad as far as i know, because of JAVA script issue.

‣ Speakers ‣ They are just computer speakers. ‣ The whole set up is to watch the TV.

BEDROOM There is nothing really electronic in my bedroom. ‣ Lamp

‣ It would be super nice. ‣ It will be one of the nicest thing to control

with my hands.

‣ It would be so nice if it turns off if i feel sleepy or fall asleep, by detecting my heart rate slowing down or something.

‣ I can still switch it off, but I don’t want to get up to reach it just to switch it off.

‣ It wakes me up while I am falling asleep. ‣ I would love to have a universal switch, or

system that turns off everything in a bigger house.

‣ BATHROOM ‣ Washing machine

‣ The nicest thing is that you can schedule it, to make it done before I wake up in the morning.

‣ I don’t want wrinkly wet clothes. ‣ It would be really nice if I don’t have to

think about the duration of washing etc at all.

‣ For example like Apple watch, it links with my alarm that I set up, and make the washing done before I wake up.

‣ It could detect when I am going to wake up, but does not necessarily have to be that advanced.

‣ The location of the knob is low so it annoys a bit when I have a back pain.

‣ The interface that is not on the eye level.

2015



BENEFICIAL SCENARIOS ‣ I can totally relate to the annoying experience of turning

all the lights off before going to sleep. ‣ I live fine with how it is now, but it existed I would be

willing to pay for the lighting system. ‣ It would be cool if I could control the my lamps at the

couch or bed, intensity and switching it on and off. ‣ It would be nice in my bedroom, to make my curtains to

go down without getting up. ‣ I don’t want to ruin the mood when making-out. ‣ Often when I am on the kitchen, I have to control the

volume of the music not to annoy the neighbours, or me. It would be nice to have.

‣ Maybe not for controlling the music only. ‣ Heaters! ‣ In spring, fall, the temperature changes a lot in

Copenhagen. When I am not here, it would be nice if they were off. Half an hour before I get home, I would like to turn it on so its warm and not freezing in the apartment.

‣ I would like my home to be prepared for myself for me. ‣ When it rains, I would like to check and close my

windows at work. ‣ Also when it is a really good weather, it gets so stuffy in

here, I would like to be able to open the windows at work, 30 minutes before - I would feel grateful.

‣ What is cool would be an optimize oven for cooking and baking, to have a bit of more information for it.

‣ Your steak is done, or your cookies are ready, to be more crispy, soft etc.

‣ The grill knows how heavy, thick the meat is. ‣ I have no idea how hot (temperature) the meats should

be, but just know the state, medium rare-well done. ‣ Also show portion, calories, allergy information etc on

my plate.

GESTURE CONTROL EXPERIENCE ‣ Nintendo Wii

‣ It is a nice idea, but it did not work precise. ‣ I do a move, but I cannot see in the screen. ‣ I did not miss it, but it is the system who did not detect. ‣ Did not feel like the natural extension of your body. ‣ I was really excited about it before purchasing, and first

time. ‣ It felt like a bit like gimmick. ‣ But it does open a lot of possibility and opportunity for

new types of games, in the world of interaction. ‣ Kinect ‣ It is just me, that was so cool, without any controller. ‣ So much better than Wii. ‣ The gym thing was cool, like having a personal trainer. ‣ I was so impressed with it. Better than I thought it

would be. ‣ I could not understand how they would do this, not

even being expensive. ‣ Reading range was a bit problematic for sports games. ‣ The dancing game was really impressive.

SMART LIVING EXPERIENCE ‣ I don’t think I have special here.

ATTITUDE TOWARD SMART LIVING ‣ I think Smart living is to make my life obviously smarter

for me, which means easier for me. Things happen automatically.

‣ Gesture control requires movement, which means that you have to ‘’BE THERE’’. It is a bit difficult to see how you are going put those two things together.

‣ I guess also some smart things also happen in real time when you are there, depends on how you think about it.

‣ It’s just funny that we need to talk about smart home, which the house has its own brain. but if i still need to do gesture control, just to turn it on and off..

‣ It is maybe just about being more and more lazy having a stove turning on and off by itself…

‣ My grandmother has difficulties in her back, and it would her really nice if there is a robot vacuum cleaner that is programmed better. Stupid robots don't work very nice.

‣ Gesture control over vacuum cleaner would be nice for my grandmother.

‣ It fits into your daily schedule, and does things ahead of time.

‣ Technology (using smart products) ‣ I am very open to Smart living, it is all about

affordability for me now - the price. ‣ I would love to have it, I am excited to see what people

come up with. They will come up with something that I have never though about.

‣ A lot of people have security cameras or gates in South Africa. My friend’s family can control the cameras and see who’s entering the house at what time from London, across the world. I think its really cool. The maid accidentally left the door open, and the family could see and told her to close the door!

‣ My grandmother has a big pain in her spine, she tries to stay as still as possible, but It is almost impossible.

‣ She is the most happy when I help her out with vacuum cleaning. She is a traditional lady who thinks cleaning is really important.

‣ She has an E-bike now, she loves it, she says its the best thing in her life.

‣ She thinks some things like just for old people. I can understand, it kinda makes you look like a disabled people.

‣ She does not like to live in the elderly house yet, but she would like to eat lunch there. Cooking, standing up for long, moving, is hard for her.

‣ Those kind of things could be automated, but maybe it is still very new.

134

2015

Respondent #03 Anders

I have my own smart system connected with my computer, all configured by me, much more personalised for my needs. I imagine it works much more smooth“

Name Anders

Age 29 years old

Primary needs: More convenience in life Biggest Pain points: Less flexibility, technological constraints

“

Nationality Danish

Living Arrangement Single / Apartment

Smart Living Experience Expert

Gesture Control Experience Expert


Household Appliances Medium


2015

LIFESTYLE



LIVINGROOM ‣ Where I spend most of time ‣ when I walk out the door(of the house), I use

voice control to turn off everything. ‣ Smart TV

‣ SmartTV with sound system, set top box, small computer running home theater, gaming device, networking stuffs

‣ Used alone to watch TV, series, gaming, and music.

‣ Music when having a party. When having a party, music is running and control it with the computer to have extra controls and also let other people to control.

‣ Controlled by remote control, computer, and iphone in distance. Never control with physical buttons on the TV (not even aware of them)

‣ I never used smart TV features, I did not even bother. because I have my own smart system connected with my computer, it is all configured by me, much more personalised for what I need. I imagine it works much more smooth than the smart TV feature.

‣ Turn on with voice control(commands) while taking of a jacket, for browsing remote control is definitely preferred over voice control(very specific commands).

‣ Voice control: Very sensitive to the noise(music, floor, street, other people), quite, home alone it will work the best

‣ Remote controller: Really easy to use, but I just have to be there.

‣ iphone remote controller: Controlling with a phone is a little bit inconvenient sometimes, it takes time to find a phone, find an app, and launch the app.

‣ but play/stop has a widget, so when TV is already running a movie, it is easy.

‣ Main Desktop computer & Laptop ‣ Also controlled by voice control, it is usually

turned on with the voice command. ‣ Use mouse and keyboard, a very standard

ways. ‣ Even when other people are around, it is

located next to the living room table so it is easy to reach and control.

‣ My computers are all connected with the server, so I can control everything with any computer, anywhere, even from outside home.

‣ Voice Controller = ‘Amy’ ‣ Speaker + TTS(Text to Speech) feedback ‣ Feedback is not conversational ‣ When I walk out of my door, I always use

the voice command to turn off everything in the living room.

‣ PHILIPS Hue ‣ Every light is connected with Philips HUE except

for a kitchen lamp, bathroom lamp, and a bedroom lamp.

‣ *Amy please, normal lights.*

‣ It can change its color, I can also hook it up with music so it blinks with the beats.

‣ BEDROOM ‣ I have my bed, and a TV in front. ‣ Philips Hue, Philips Hue hub, A lamp detached

from Philips Hue, Electronic Piano, Classic TV and set top box, Philips Wakeup lamp, Wireless router.

‣ I guess I don’t use my ceiling lamp that often. Physical switch does not bother me much since I don’t use often.

‣ Voice control is not used, because Amy cannot hear me. I just use my phone to control all the light before sleep.

‣ A button on Philips Hue hub is only for paring, one time use, It is like a setup thing. Some sort of security measure.

‣ Philips Wakeup Lamp ‣ I had to for many years, so it does not work for

me anymore. ‣ TV and Set top box ‣ TV here, in the bedroom is only controlled with

remote. ‣ I want to connect it with a computer to see more

contents.


135


2015

LIFESTYLE



KITCHEN ‣ My kitchen is the most un-techie place, and I

would love to preheat the over with my phone, or with voice control, adjust the stove etc.

‣ Oven Stove / Ventilation ‣ Traditional, and non-techie. ‣ The only inconvenience is that I need to

come over to preheat. just the matter of get up and get in here, without having to put anything in the oven yet.

‣ Dishwasher ‣ Just a knob, and buttons ‣ Fine for my use. ‣ No remote control, or other technology is

needed. ‣ I just need to fill, and turn on. it would be

really nice if it could fill itself. ‣ Timer is needed.

‣ Microwave oven ‣ Not really techie either, it just has a

standard digital display, push button, timer knob.

‣ I have to get in here and put the food in anyways, so having a knob is kind of fine.

‣ I would more interested in a feedback, timer, on my phone or something.

‣ Refrigerator + Freezer ‣ I don’t get my hands messy during cooking,

so that is a no problem for me although I can see other other people might have a problem there.

‣ Oven

‣ I don't have a problem ‣ BATHROOM ‣ The light is not connected with Philips. ‣ It does not make sense to control the light outside

of the bathroom, If I have to be in there anyways, and also when someone else is in the bathroom.

‣ It could just be a motion sensor, you just use the light only when you are in the bath room.

2015



BENEFICIAL SCENARIOS ‣ I can’t always use voice control when there is

noise around, so other types of control, gesture control might be useful.

‣ Situation when I cannot use my voice control, like when I have music for a party, gesture control can be a good way to make up for that.

‣ Using it to change TV channels, but of course you have a remote right here within reach, so why would you.

‣ Its the matter of what is the most convenience.

GESTURE CONTROL EXPERIENCE ‣ Kinect/Xbox ‣ I used it for work, we used it for custom build

apps, usually for games and entertainment. ‣ It worked perfect, since it does not have a lot of

sensors attached, and people liked it. ‣ Voice control ‣ I have used it for 3-4 months. ‣ I am really satisfied and impressed, and I use it

everyday. When it is quite, it works really good. ‣ Voice control is used mostly when I come home,

and when I want to turn on stuffs. ‣ I also used voice control sometimes to control my

lights more specifically, the colors of the lights. ‣ To play Specific movies or artists. ‣ Just very good not for browsing, then you have to

talk to much. It is not convenient anymore. ‣ Gesture control is all about convenience. Voice

control is not always possible.

SMART LIVING EXPERIENCE ‣ Philips HUE ‣ It did not meet my expectation because of API

(different scenes not getting deleted from the server but only from the app. No full control over the scene system.)

‣ Basic use is satisfying. ‣ Smart living is kind of what I already have for me, ‣ To me, smart living is just telling your house what

you want to happen. ‣ What I really see is an AI that understands you,

with much more advanced microphone that differentiates commands and noise, different people, your location, just much more AI.

‣ The focus is convenience.

ATTITUDE TOWARD SMART LIVING ‣ Technology ‣ Cloud based AI services: Google, microsoft,

Wipro has a service coming.. ‣ Which I would see it can be connected with Voice

control. ‣ Machine learning, understanding, interprets and

responses. learning from people and the internet. ‣ I would definitely accept those technologies. I am

already one of the kind to accept those… I will do is as soon as possible.

‣ My study definitely influenced me, but I also see my non-techie friends impressed with the voice control technology and they say they would like to have it.

136

2015

Respondent #04 Philip

Smart living is like your own butler. It knows what you want, what it needs to do for you, more automated, intelligent, and contextual.“

Name Philip

Age 30 years old

Primary needs: Connected home experience, in and out Biggest Pain points: Inconvenient steps, usability, aesthetics

“

Nationality Swedish

Living Arrangement Nuclear family / House (9 months boy)




Household Appliances Many


2015

LIFESTYLE



LIVINGROOM ‣ Eating, sitting and talking, playing boardgames,

and then just getting around the TV, but not with the TV on. We don’t watch TV much to entertain.

‣ Main TV ‣ Nowadays, we don’t spend most of the

times in front of the TV anymore, but around the dining table and then the couch.

‣ I use a remote control, among physical buttons, remote control and my phone.

‣ I control my TV with my smart phone as well. it is easier to see the overview.

‣ For TV, the remote control is the easiest. I don’t always have my phone around, but the remote control should be by the TV.

‣ Flexible stationary. the proximity of it. ‣ TV remote control is supposed to be near

the TV, when it is not, it is annoying. ‣ IR sensor needs to be pointed at the TV is

stupid. ‣ The TV has a motion sensor. If I am not

around, the screen is turned off, and the sound is still on. It saves energy, it is good.

‣ The buttons are good, but they are not nice buttons, squishy, and not a lot of feedback.

‣ Apple TV ‣ Sonos sound system

‣ There are three buttons, for the volumes and play/stop.

‣ I don’t run to the speaker, it’s easier to reach my phone, unless I have a phone call.

‣ Philips HUE

‣ At least every light in the living room will be connected with Philips HUE.

‣ Smart plugs ‣ It is connected to network, you can control

over your phone, to turn it on/off, set a schedule, even when you are away from home.

‣ I would like to connect things like an Iron. ‣ It is more for the security. ‣ Especially on a vacation, I can pretend

somebody’s home.


137


2015

LIFESTYLE



‣ KITCHEN ‣ Mostly cooking, but also socialising, and just

hanging out. We do have a lot of guests, so we hang around here while cooking, have coffees. we gather around here and the living room.

‣ Stovetop ‣ It works with touch, without any hectic

feedback. ‣ Controlling it with a remote could be cool. ‣ When we have wet hands, when chopping

vegetables, it does not work properly. ‣ Range hood

‣ We have a ventilation system that has a remote. It is the only way we control it.

‣ IR sensor could be locked if we start to have a lot of things on this table.

‣ I might mount the remote on the wall. ‣ Microwave

‣ we just use simple function, quick start. ‣ It is a bit hard to maintain the door open

with one hand while holding Oliver. but it is not very hard.

‣ Oven ‣ Oven is quite basic, just with the knobs. ‣ The knobs are pretty precise, I like the

tactility of the physical turn and the direct feedback.

‣ I probably prefer a physical knob than a touch interface.

‣ I don’t like messy fingerprints on a high glossy surface. It could have been a matt surface.

‣ I wouldn’t use the oven while I am holding Oliver, because it is too hot.

‣ Coffee machine ‣ I actually like touch buttons on this one,

because I don’t like usual Nespresso buttons - squishy and rubbery.

‣ I like the combination of physical and digital feeling.

‣ During the night, the light from the buttons actually makes the icons invisible.

‣ Heater for floor heating ‣ It is touch, you don’t get any sound or

tactile feedback, you only see the numbers from the screen.

‣ If it were automatic, it would be cool. It is hideous to come over just to control the heating system.

‣ There could be a remote control, but where would we have to put it then? its too much for such a simple task.

‣ Voice control would be nice ideally. ‣ Lamp for cooking top

‣ It has a remote control, and it is the only way.

‣ We are thinking of fixing it more stationary. ‣ I will only use this lamp when I am here. ‣ I don’t need the control to be mobile. If I am

here, I need the light to be here, or it could be like general motion or pressure sensor.

2015

LIFESTYLE



‣ HALLWAY ‣ Security Camera

‣ We don’t control it, but it only detects break-Ins.

‣ It is turned on when we lock the doors. ‣ It will send pictures to the security

company. ‣ It has an app, so we can control the

security system and the smart plugs. ‣ Floor heating for the hallway

‣ Physical turning knob, and on and off switch.

‣ This is the bad kind of knob, hard to reach, read and not precise (no clicking).

‣ We just set this on once and don’t interact a lot with it, maybe more during winter.

‣ Since this is not time critical, it is also not a critical thing to adjust, there are not so many critical problems.

‣ Baby monitor ‣ Baby monitor parent unit is located in the

hallway. ‣ Somethings have too many steps, volume

adjustment is what we use the most, but camera adjustment is set to the first function.

‣ We would sit in the couch, yard, kitchen, and we will have the monitor to watch Oliver.

‣ It is not really good, but it is okay. it has a bit of usability problem.

‣ GUESTROOM ‣ Does not have much.

‣ BABYROOM ‣ This room does not have much either yet. ‣ Just a light dimmer. ‣ He is not in this space often yet. His

jumping thing is here.

‣ BATHROOM ‣ Towel heater

‣ It is nice to have a dried, warm towel. ‣ Thermostat

‣ This one is better since you can set a schedule.

‣ But it is not connected with my phone.. ‣ Because of its button system, I have to go

through a lot of steps. ‣ Ventilation system

‣ Automatic, it senses humidity level.

‣ BEDROOM ‣ We don’t have a lot of electronic system here. We

are just planning to connect with HUE light. ‣ Babybed

‣ There is a baby camera at the corner. ‣ We use it often, but don’t control it often.

we just put it there and watch it with the remote control.

‣ Temporary light ‣ It is annoying to bent down, and turn it on

and off.

‣ We would like to set up the time with HUE light and have some regularity. Also like a wake up light.

‣ BATHROOM

‣ BASEMENT ‣ Planning to set a home theater in the future. ‣ Washing machine ‣ Dryer ‣ Iron ‣ Control Pool system

‣ Thermostat, etc ‣ Hard to come downstairs only to control the

pool system. ‣ Light

‣ Motion sensing would be nice.

‣ BATHROOM

‣ GARDEN ‣ Robot vacuum cleaner for the pool

‣ We just throw it in the pool, and it just goes by itself.

‣ As much as automation is awesome here. ‣ If there has to be more interaction, i would

like to have nicer buttons. ‣ Touch screen sometimes does not work

with high glossy surface. Its always a fight between aesthetics and more functions.

138

2015



BENEFICIAL SCENARIOS ‣ A lot it the kitchen, when you mess up with your

hands. ‣ Give more tactility. ‣ Remote control to adjust the shades, it would be

nice it could have sensors. ‣ or even more remote. So I can control at work

when I know it will get too hot at home.

GESTURE CONTROL EXPERIENCE ‣ Kinect/Xbox

‣ One of the better gesture control device I have every used. just precise, compared to wii etc.

‣ Leapmotion ‣ It is good since it is mobile. ‣ If I would have one, I would like to use it for

3D model. ‣ Not effectively modelling, but when

showing to the client, if they could navigate by themselves, instead of asking me to turn around will be nice.

SMART LIVING EXPERIENCE ‣ I like the automation. ‣ Less things to think about. ‣ Focusing once, and it works, and it should work.

you don’t have to remember or think about it anymore.

‣ Don’t have to remember turning on/off, heat, temperature

‣ convenience.

‣ I actually have low expectation so I am pretty satisfied with what I have so far.

‣ HUE light in terms of light quality is good, but Philips app is bad.

ATTITUDE TOWARD SMART LIVING ‣ Smart living is like your own butler. ‣ It knows what you want, what it needs to do for

you. ‣ More automated, more intelligent, contextual. ‣ The time of the day, knows when and where I am,

and when I will be home, just searches information and predict what I am looking for.

‣ Assistance.

‣ Technology ‣ Cool. I just love it, to see what is possible. ‣ Its also fun to see what does work and what does

not work. ‣ I will definitely use technologies that makes my

life more comfortable, if it is affordable. ‣ I would buy whatever if I were living alone.

2015

Respondent #05 Camilla

The girls actually really like helping out, so it would be nice they can be more involved. It is nice to do together. I just wish it was a little bit more easy and smart.“

Name Camilla

Age 37 years old

Primary needs: A bit of more convenience in life Biggest Pain points: Too much house chores to be done

“

Nationality Danish

Living Arrangement House / Nuclear Family


Gesture Control Experience Novice

Technology Acceptance Conservative




139


2015

LIFESTYLE



LIVINGROOM ‣ We just moved in, so everyone is living in the

living room while the upstairs are getting renovated.

‣ TV with set top boxes ‣ We would like to switch it off always rather

than stand-by, to save energy. ‣ We have a smart(master) plug, if the

tv(master) is turned off, everything gets turned off.

‣ We have 3 remote controls. We use TV remote control only to select sources, and adjust volume. another remote control is used the most frequently, to control the TV box, different TV channels, start and select Netflix. The stupid thing is that it has a volume button as well, but TV volume(the first remote) is a master volume so it needs to be controlled as well. The other remote control is for the DVD box.

‣ I would pick up two remote controls together, to control the TV and TV box, and have it besides me.

‣ We still have DVDs, the girls really like to have them physically, looking through the CDs. We will also bring a small DVD screen when going on a trip in a car. Netflix is difficult for the girls to control, because it always recommends what to watch. and it is also kind of in the same flow all the time.

‣ I think it is a little bit annoying that you really have to point directly at the box to make it change.

‣ The small one does not know the flow of the remote control. She would interact with the TV as if it were an iPad with a big screen.

‣ My parents in law find it really difficult to control the TV set when they are here to babysit. I help them how to control the system by telling them step by step on the phone. I even had to do FaceTime when they were totally blank last time. It is not a catastrophe, but it is just annoying.

‣ iPads ‣ Of course we have iPads, so both girls can

start Netflix on their iPads if the TV does not work, no problem. and they can watch separate usually they want to watch different things. They can see Chromecast, Podcast, TV, Netflix, games, sound books.

‣ We(Camilla and her husband) have an iPad as well.

‣ Philips HUE Lamp ‣ Its a bed time lamp for the girls, and its

connected only with her(the big girl) iPad. It is her lamp, and she has no problem controlling it.

‣ They really like the interaction, to change the colors and plug it in to music so it is like a disco. We might want to buy another one for the small one as well.

‣ If they forget to turn it off, i will just find her iPad to turn it off. but it is usually on during the night, since its is the night lamp for the girls.

‣ NightLamp ‣ We have a night-lamp that is turned on when it is

dark to show where the toilet is. It is triggered by the light level, so when it is dark in the evening, it is always old.

‣ Robot vacuum cleaner ‣ We bought them, but not set up yet. ‣ I am looking forward to it!

2015

LIFESTYLE



KITCHEN ‣ Fridge (without a freezer)

‣ It is messy. ‣ I am the only organiser of the fridge. ‣ One of the issue is that things get expire,

and i am the only one who cleans it, and if i don’t have time for it, those will just stay there.

‣ Dishwasher ‣ It runs once a day. We usually start in the

morning and collect the dishes in the afternoon. We always use energy saving mode, it takes 2:17 hours. The time is not an issue.

‣ In the morning, the dishes don’t get in to the dishwasher, but just piles up on the table top or sink.

‣ Its either me or my husband who operates it, We are trying to teach the girls to put the dirty dishes in the dishwasher.

‣ It would be nice if it had an indication that dishes inside are clean.

‣ Sometimes the clean dishes stays in the dishwasher for a while, and the dirty ones pile up on the table top.

‣ It is so quite and you don’t hear that it is actually on.

‣ Controlling a dishwasher is easy.

‣ Induction Stove (cooker) ‣ It is quite good, it is quite fast.

‣ You can lock it, so it doesn’t start up. It is nice that you have to activate it, especially when you have children.

‣ It is finger controlled, it is just touch. ‣ When you start it, it starts summing. ‣ It has a timer as well, it makes a beep once

it is done. It is really nice that it switches off itself when it is done. I use the timer for cooking eggs or potato, when I know the certain amount of minutes.

‣ Also with induction it won’t work without a pot. (safety)

‣ The girls are not allowed to use the stove yet. They love to join cooking, helping out in the kitchen standing on a stool or sitting on a tabletop, but I have a rule that they are not allowed to touch anything on the stove, pots with boiling water for example.

‣ Ventilation system ‣ There are buttons for light, and intensity, I

don’t know about what other buttons are for.

‣ It could be nice when you don’t need to push the buttons for the intensity when you are cooking four things at the same time. It would be nice if it can detect the humidity. When I am in the living room, I see smokes, and moist, it could be nice if the ventilation could regulate it automatically.

‣ Radio ‣ We will listen to the radio all the time, in the

kitchen, and also in the living room.

‣ I stick to one channel, and my husband sticks to another. During the weekend, we have an argument to listen to what we want.

‣ It could be nice to control the volume without having to come here.

‣ It is played all the time, we will switch it on once, leave it, and switch it off in the night.

‣ Sometimes we need to switch off when Danish news is on the radio, and I don’t want my daughters to listen to it since she is too young for it.

‣ Microwave ‣ We only use it for reheating and defrosting.

‣ Oven ‣ I haven’t tried setting a timer yet, and I

usually don’t set a timer, just see the clock and cook by a hunch.

‣ I start the oven, and it gives a beep when it is preheated. Sometimes I just start cooking without preheating.

‣ Quite often, I forget to switch it off, after cooking food. I realize while we are eating, because I can here the sound, and the light.

‣ When the food is cooked and starts to smell, it is quite hectic the girls asking when they are eating etc, rushing to the table etc, there are a lot of things happening. so I only get to realize when we settle down on the table to eat. I sit close to the oven so it is easier for me to interact with the oven, easy to come back to eat.

140

2015

LIFESTYLE



‣ Lighting switches ‣ We just switch is on and off. ‣ They are switches and dimmers. we don’t use

dimmers. it is a bit 90s. ‣ I think in Denmark, most of people are trained to

switch off the light automatically when you leave. when I was a child, it was a big thing to save energy because it is expensive. I think just by default, you turn off the light when you leave the room.

‣ Refrigerator ‣ We just open it to use. ‣ When open too long, it warns for temperature rise. ‣ I can change the temperature. If ii needs to cool

really fast, I can press this button. ‣ The suction makes it sometimes hard to open the

door. Especially the girls have hard time to open the door, they found their way to stick their finger in to release the suction.

‣ The door is opening the wrong way for our setting(pathway).

‣ It could be nice to open the door from the both ways, especially in an open space like this.

‣ I would like to use some part of it as a freezer, especially for summer, for ice cubes etc.

‣ We have a freezer in the basement. ‣ I would like to have a two doored fridge with ice

dispenser, but their energy efficiencies are low, it uses a lot of energy. I consider those energy grades, since the energy is pretty expensive. you use home appliances everyday, all year, and it is a lot.

BASEMENT ‣ Here we actually have a light that is triggered by

movement. This is just from IKEA. The good thing is it is on batteries. When you come down to the basement the light just turns on and you don’t forget to turn it off.

‣ Washing room ‣ Washing machine ‣ Clean clothes for each of the family members ‣ Dirty clothes are collected here and also from the

upstairs. Two piles depending on the temperatures. ‣ We just have to buckets identical in both levels,

because we take off our clothes upstairs. ‣ I only use two programs, either 60 / 40. ‣ Mayby wool function.. for baby clothes when the

girls were young. ‣ Washing machine has too many functions for me I

don’t like I have to go through all the different functions(knob) to set a function. I would like what i use to be default option.

‣ I would start the machine and go away. it takes like forever, … 2 hours.

‣ I know when it is done, I know it by instinct. I would forget sometimes. but it is not a big issue since we wash our clothes often, so it wouldn’t be forgotten for a long time. I don’t like the beeping sound. I like our dishwasher since it is really quite. I especially hate the beep during the night. it is stressing you, give pressure, a little bit

‣ Dryer ‣ It is really really low-tech, you set the temperature,

and the time by yourself. ‣ This one lacks the function to measure if the clothes

are dried. You come down when it is done, and the clothes are still wet(damp).

‣ I don’t have any trouble with the knob itself. ‣ There is a big hurdle of sorting the clean clothes to

put into the closet, that is awful, takes a lot of time. we have baskets for clean clothes for each, and I would put everything back in the closet only when they are full. I cannot go up and down all the time.

‣ Unless the drying is done, there is no other interaction in the middle.

‣ Getting it down here with dirty clothes is okay, but all the sorting with clean clothes is annoying. We are 4, and its usually my job, my husband have a problems sorting the girl’s clothes.

‣ It could be nice if some kind of RFID can be there and it knows which clothes belongs to which.

‣ we should put it into a washing bag, Alma puts it in a bag, and it would be sorted afterwards, a lot of sorting bags, we cannot fill up the machine totally. that is a waste of energy and water.

‣ I have pair-less socks, that is a big issue as well. ‣ Freezer ‣ Really old freezer, not ordinary. ‣ It is just on/off, hardly any interface. ‣ It is deep so it is hard for the girls to reach. ‣ It could be nice to have an overview of what we

have in the freezer. ‣ Heating and Electricity ‣ I think we have different settings for it, which I

haven’t tried. It can be programmed that the heating goes up and down during the day and year, you can also control the water temperature in the shower.

‣ Actually it could be nice that the water temperature can be voice controlled.

‣ We are four people in the house, and we all prefer different water temperature.

‣ Now it is me, please adjust the temperature to ‘XX’, without having to turn the knob all the time. it is usually set to the last person who used the shower.

2015

BENEFICIAL SCENARIOS

‣ There are lots of activities, if it could support some of the actives at home

‣ Taking off the clothes, putting the bags at the right position, the light turns on and off at those situations

‣ bed time, TV, to be able to control. ‣ They use iPads to watch things, and I tell them to go

to sleep after watching a program, but I can never check if they do so. It always slips, so having a little bit of control of their iPads would be nice. I could turn off in 10 minuets, instead of go in and put on an alarm..

‣ I would like it if I say turn off in 10 minutes to my girls, and the whole system understands it and turns the iPad off 10 minutes later (I am Master controlling the house).

‣ Heating in the shower. If could identify whether it is me or not.

‣ I wish it was really easy to set up. The set up is really hard, program anything etc. I don’t like doing a lot of things to set up to program, before using it.

‣ If I have to use gesture control, it needs to execute the default function that I use every day. If i don’t use everyday, it is okay to set up and program.

‣ It would be nice if the girls can have more control in the appliances, instead of navigating a remote control interface etc. I would like to give the girls more flexibility in controlling things, if it is safe.

GESTURE CONTROL EXPERIENCE

‣ Wii ‣ I only know it for playing. I tried the tennis. ‣ I really liked that it was kind of a social experience,

involving all the participants, much more than a screen.

‣ I actually think iPad is really good for quite games, entertainment.

‣ But for Wii, you get the movement as well. you sweat, it is more interactive, healthy way of playing.

‣ I assume that the girls would be more engaged in controlling it, since it is just swiping etc.

‣ The small girl would use the TV as if it were a big iPad, and do gestures in front of the TV.

SMART LIVING EXPERIENCE ‣ Security ‣ That should be a smart system which i can watch over

phones, etc, since we have a house now there is a bigger tendency of being robbed.

‣ I want to monitor even when I am at work, just everyday life. But only for the intruders, I wouldn’t want to know whom in the family enters the house. I don’t need a security for my father or my daughters if they are at home on time. I think it brings mistrust in family.

ATTITUDE TOWARD SMART LIVING

‣ I think until now, the focus has been on features and technologies, I would really like it if it had more to do with the needs I would see the benefits, then I would probably buy it.

‣ There are some issues to be more smart, like heating system, electricity,

‣ Especially for a fridge, to organise and see what’s in it.

‣ Clothes, there are so much. the seasons are changing and i have to find them and i cannot remember at all. Going through the winter clothes takes a lot of time , it is always overwhelming so I postpone till the winter is here.

‣ Efficiency and being smart is important, only for very extreme tasks for me.

‣ It is a part of the family to maintain the house. It is a job we take, and do together. It is also a part of the upbringing, how the household works. If everything is done when we come home, its feels a bit less home. it is okay if i need to vacuum, wash the clothes, but just wish it was a little bit more easy and smart.

‣ The girls actually really like helping out, so it would be nice they can be more involved.

‣ It’s nice to do it together. ‣ Acceptance ‣ We don’t use that much money in technology. It is a

bit off for us to have three iPads. ‣ I need to be convinced that i can get benefit from,

then I would buy. Otherwise, I don’t investigate too much to it. if we don’t get convinced with the feature strongly, we will just buy a cheaper one. I just need basic tasks.




141


2015

Respondent #06 Katrine & Michael

Everyday life being more convenient. You are able to download different levels, instead of starting from scratch. “

Name Katrine Michael

Age 36 years old 40 years old Son (4 yrs old)

Primary needs: Connected, and shared home experience Biggest Pain points: Forgetting tasks, duplicated tasks

“

Nationality Danish

Living Arrangement Nuclear Family / House




Household Appliances Many


2015

LIFESTYLE



LIVINGROOM ‣ We would spend most of the time in the living

room. ‣ I(M) set the interfaces, and she(K) uses them. ‣ TV

‣ Connected with Apple TV. ‣ Raspberry Pie ‣ We use it to stream, watch films basically. It

is quite hard to set, we had someone else. ‣ We hardly use it. ‣ (K) I bought all these stuffs, but I don’t

know how it works. Every birthday he wishes for something like this.

‣ Philips HUE ‣ Anytime when people coming over, we use

it. ‣ During the night, winter, warm light. ‣ Control on the phone. ‣ You can do it with a remote control as well.

‣ Logitek remote control ‣ It is universal, so we use this one. it is

programmed to use with HUE, TV, Apple TV. ‣ We used to have several remote controls,

so many. ‣ I know it is programmable, I haven’t

finished it yet. It is a bit technical. I only did it with the TV. so I just use the app.

‣ I use the remote for TV when it is reachable. ‣ (K) It is pretty complex. I don't even know

how to control the lights with this. ‣ My parents, could not do anything but

turning on the TV and off.

‣ They still only can use simple features. ‣ This is a touch screen, with physical

buttons, it is weird. two different input ways. High tech touch screen - old time physical buttons does not just make sense.

‣ I only want one thing at a time, a touch screen. So I use my app.

‣ We don’t use smart TV features. you could download apps and use them.

‣ I don't go into smart TV interface at all. i don't go into an internet with a remote control.

‣ It is not really user friendly. ‣ I might do it with voice control. ‣ There was a lot of different options, large

screen (remote control), it was always always out of battery which was so annoying.

‣ (M) I try different things. ‣ (K) I will never read a manual. as

convenient, intuitive as possible. Different level, easy level to normal complexity, dive into different layers, because then it would be adjustable.

‣ Then you could share whatever box, TV, remote, you can just select whatever the level you want.

‣ Tone of boxes: Apple TV, battery charger, router, switch, satellite receiver, Logitek harmony, etc. We only open this to charge something. we always use remote control.

‣ Sonos

‣ It is amazing. ‣ We control with phones and ipads. ‣ We have done programming with Logitek

again, it was complex. ‣ Sometimes they change the interface, and

you get annoyed by it because you use it every day.

‣ Because everybody can control with their phone, we could have a music quiz to guess songs and artists.

‣ We made that game, from and using user interface so quick.

‣ I will use physical buttons only when my phone rings, when we are eating and hard to locate the phones.

142

2015

LIFESTYLE



BEDROOM We don’t have a lot of devices here, it feels uncomfortable to have a lot of things on your head. ‣ Sleeping device

‣ (M) I would like to buy a new one. I have tried with an app, it was fun. I’ve been looking for one from US, which tracks movement, heart rate, breath per minute, sleep phase, etc.

‣ I really like passive monitoring. I don't want to sleep with wearables. I felt like a miner to wear something on my head during sleep.

‣ I want it to be as passive as possible. ‣ and integrated with my habits. ‣ I want it to automatically detect when I am

going to sleep etc. ‣ automation is an important feature.

‣ Tones of chargers ‣ Fucking freaking codes.. ‣ Your bed is so filled with wires. ‣ It is basically all the same, identical, so at

least not confusing, but still many in numbers.

‣ Lamp ‣ It is not connected with Philips HUE (which

is only for living room). ‣ I’d like to use my lamps with voice control

or gesture control, not to get up from the bed.

‣ It may sound a bit lazy, but it is about convenience.

KITCHEN ‣ Oven

‣ We use it often, like every second day. ‣ I would very much like to use voice control

or gesture control. ‣ When Katrine is out, she sometimes gives

me a text message like: remember to switch on the oven. If she could do that from a distance, it would be very convenient.

‣ (K) A lot of our meals are delayed by waiting.

‣ I come home fairly late, but we have to eat on time because of our son.

‣ Fridge ‣ (M) When I am into training, I like to track

calories. I have an app, but I’d like to have it with my fridge, combined with the fridge without having to approach other thing.

‣ It will have to come with inventory tracking. ‣ considerably reducing food waste as well. ‣ (K) For me it is more convenient to have it

on the phone. I am at work. I cannot be here.

‣ (M) No, I would like to have extra interface, connected with Wifi, and a cloud service, we could use a fridge to plan. It shows expiry dates, not only on your phone, but right here.

‣ It is normal for you to check out what is in there here.

‣ If I buy a new fridge, I would definitely go for app integration, or smart component.

‣ Coffee machine ‣ We have a lot of different ones, but none of

them are intelligent. ‣ Induction cooker

‣ (K)We just turn it on ‣ I think it has a timer, but I never used it. I

almost never turn it on, I know when it done. or when I am not sure, I set a timer on my phone.

‣ I always get my phone. ‣ It is not really inconvenient. ‣ (M) Sometimes it is a bit annoying that you

can only set one timer in an iPhone. ‣ I would like have a kitchen computer that I

can talk to. ‣ Your whole kitchen will be so connected

that would be amazing.

2015

LIFESTYLE



BATHROOM ‣ Scale

‣ Connected with an app ‣ You step on it, and you get a notification on

the app wherever you use. ‣ It monitors your health in a way. ‣ It is much more than just a scale. ‣ It is connected with Wifi, Tracks activity,

shows a dashboard… ‣ (M) I use it 5 days a week, Monday to Friday

to check the progress with my cardio training. I am satisfied with it. Sometimes it takes time to calculate.

‣ It has an air pollution sensor in it, and also in the app as well, so it will let you know of the weather today.

‣ I am very excited about the app features and that it is connected, and when you step on a scale, you get a weather forecast, you get an idea that it is going to rain today. it is integrated into another process, automatically you get the weather data without having to check your phone to give me an idea to work out inside or outside.

KIDS ROOM ‣ CD player

‣ This is the most advance device in this room.

‣ The kid does not have a phone yet, so this is the only way he can listen to music by himself.

‣ iPad ‣ He has one iPad. ‣ He wouldn't control things in this house yet,

we haven't tried actually. Maybe if Sonos had an app for kids, soft app.

‣ (he is still in a stage to learn how to spell his name, Katrine’s name etc)

GUESTROOM ‣ -

EXERCISE ROOM ‣ Heart-rate monitor

‣ I have a heart rate monitor, Wahoo X (Wahoo Tickr X), if I just press my chest it checks my heart rate. It is awesome.

‣ I could use it for my bike, which has a power meter on.

‣ It gives me data that syncs with Apple health. It syncs with my weight gain.

‣ I can see my aerobic activity, and how it affects my weight and fat mass.

‣ It’s more integrated. ‣ I really like the health perspective, because

I am a doctor. ‣ I have different systems that talks together,

if you have a data combination it is pretty cool. I really love system talking to each other.

‣ I really love to see the exact data. When you just looking at the stats, steps, heart-rate, the amount of calories burnt, it is pretty cool. ‘oh, I pushed myself hard today than yesterday.’

‣ (K) he has a real interest in connecting everything. Life style, health, including exercising, together with food, sleep.

‣ (M) There are so many sub-systems. I’d like it to be automatic. It takes some efforts to get into it.


143


2015



BENEFICIAL SCENARIOS ‣ Our phones are often out of power. We have a big problem

controlling things when our phones are dead. ‣ Normally our phones are here (on the kitchen table) as soon

as we are back home. ‣ Voice control / Gesture control ‣ With voice control, you could control curtains, or doing

certain tasks. ‣ I am very much looking forward to Apple home integration.

SIRI and phone integration, just make your phone to do it. ‣ I’ve been looking for it for years and they are still early in

the phase, and I don’t want to purchase it yet. so just wait-. ‣ We did the same for the HUE light. ‣ Definitely the music. not the selection of the songs, but

definitely the volumes. ‣ Dimming the light. ‣ (K) Still very simple stuffs, but they are what I do the most,

turning on the music and controlling the volumes. ‣ (M) I believe in Voice control. ‣ (M) NFC. you could program some activities, all of my

coffee machine go and make coffee for me. automatically activate the coffee machine.

GESTURE CONTROL EXPERIENCE ‣ (K) We don’t have any gaming device here. If we had it, it

will make our son more difficult to play outside. ‣ (M) but with gesture control, you can actually make him to

do physical stuffs. ‣ (K) okay. depends on the game. ‣ No experience with Kinect/Xbox, Leapmotion, MYO band.

SMART LIVING EXPERIENCE ‣ Apple watch ‣ (K) The HUE control on apple watch will be an option. I

wear it all the time. When sleeping, I will keep it besides the bed.

‣ (M) I am not buying one yet, too clumsy. too connected with the phone.

‣ Smart thermostat ‣ We want thermostats that are connected with each other.

We actually wanted NEST. It was not compatible with our system, Danfoss. I am a bit annoyed about that!

‣ Baby monitor ‣ We wanted to buy a sound (baby) monitor, when the son

had a breathing problem. ‣ Home Security ‣ We are unsure if all the doors are closed, when we are

leaving. we would like to know. ‣ I would like have an electronic lock. When I come home, I

hope it knows it is me, and I can just walk in. Especially when you have kids, your hands are full, and then you have to look for the key, and have to put something down, it is so so annoying.

‣ The area is pretty safe though. ‣ Communication ‣ (M) I want cameras and microphones in the living room. I

would like to watch family, have family time while away from home. (M) FaceTime, you have to agree with time, its two-way.

‣ (K) We have FaceTime for that. I am not really into that cameras. It is a bit creepy.

‣ EXPECTATION ‣ (K) Sonos is simply amazing and great, apart from the fact

that I don’t know how to set it up. Just because I am lazy, and I haven’t tried yet.

‣ Sonos work perfectly, Philips HUE works really awesome especially during winter.

‣ (K) I could get a bedside lamp controlled with iPad from our son. The light is actually one of the things that Strøm loves to try. He tried a couple of times, and he loves it.

‣ (M) The scale, I use it everyday, and its great. ‣ (K) The functionality is great, but I see tons of different

things in the app, options to elaborate in the system. ‣ I feel like I need to sync all my weight, physical activity, etc,

and I was actually hoping that Apple watch could be proven to do that. but I’ve tried that but wasn't that simple.

‣ Those kind of things - leaves me the feeling of I should be able to do more with that.

‣ (M) I see my heart rate monitor precisely measuring burnt calories in the training room, I see my weight a day after. If you have a good habit of exercising, when you really see the improvement, it is SO MOTIVATING.

‣ connecting feature with the weight etc.

ATTITUDE TOWARD SMART LIVING ‣ Our main point of having smart technology at home is to be

convenient. ‣ (K) Convenience of every data. ex. Grocery shopping,

recipes, cooking, controlling the oven, but also like family planning, calendar (expire dates as well!), weekly plan.

‣ From time to time, we argue who promised to pick up our son.

‣ (M) Everyday life being more convenient, and the whole health perspective. Motivation is essential to make your habit change.

‣ Acceptance ‣ (K) I use the most simple, intuitive ones. Simple Philips

HUE, TV stuffs that are already programmed. ‣ It should be something that you are able to download

different levels, borrow from what other people uploaded, instead of starting from scratch.

‣ (M) A lot of different factors, usability, intuitiveness, easy to use. Wide approval factor - cool design, to make it cool!

‣ (K) I am a designer so I need to consider those kind of stuffs. I like everything hidden. Able to be hidden, or small enough.

‣ (M) It has to prove that it makes impact especially in the early phase. I will buy it anyways because I am a nerd, I am interested in it, but if my parents, or my son were using, that has to make an impact, and be simple.

‣ Installation is the key. ‣ Technology

2015

Respondent #07 Tommy

Having kids and family is about life quality, gadgets will get you more disconnected with your family.“

Name Tommy

Age 40 years old

Primary needs: Comfortable assist in an ambient way Biggest Pain points: Intrusiveness, distraction, maintenance

“

Nationality Danish

Living Arrangement Nuclear family / House






144

2015

LIFESTYLE



‣ I live with my wife, Lisbeth, 14 year old son Christopher, and 11-year-old Manuce,

LIVINGROOM ‣ We have a living with a couch, iMac, computer, iPads,

connected with dining room and Kitchen, I like that the space is connected all together.

‣ TV ‣ The kids are almost out of the interest, they are

more into their computers and iPads. I and Lisabeth are more into TV.

‣ Most of time, we watch TV on a couch, but sometimes it is just on like a radio.

‣ We got a Mac mini for apple center, apple TV for Netflix, and we got mediaflex, and Netflix. the small package.

‣ We did not get a sonos speaker. I never got to see them..

‣ The kids are more into Youtube channels, also weird stuffs. Normal people who broadcasts are more of their interests.

‣ On Friday night, we watch movies together. ‣ Simple, TV has a remote, and the app TV has a

remote, and Mac mini can be controlled by a remote, but also by a keyboard and mouse.

‣ I did not want to have a smart TV actually, so we had to have a Mac Mini, because I could control it better, I know how it works, and I did not know what Smart TV really was.

‣ Mac mini, I’d like a computer to be attached, because I think smart TVs are somehow limited. In terms of finding the right contents. and they are kind of stupid, you cannot upgrade it etc, the technology seems a bit immature. Maybe the new ones might be possible to upgrade.

‣ Controlling with a remote is okay, compared to really hard stuffs. It haven't been evolved for like 20 years. It is a classic, it works, and you don’t have to recharge often, yet.

‣ I hate running around to charge stuffs, to have it on all the time.

‣ I have a small desk for the remotes and set boxes, so they don’t get lost.

‣ Light ‣ This and that light are connected. It is more for

the ambience, coziness, during the night. ‣ I never use the switch on the floor, but just on

the wall. ‣ It is simple to use the switch on the floor still. ‣ If I forget about switching off the light, I would

come downstairs to turn it off. I am not afraid of it. We should move more anyways.

‣ Moving and stretching over are no problem. ‣ iMAC ‣ It has a large screen, It is easy to use, It is

connected with spotify to play music etc. ‣ The router could be turned off very occasionally. ‣ My kids late for dinner because of gaming is not

acceptable. It is a family time. We’ve got hot food, we don't want to spend time making good food when no one is there. It needs to be respected. It is not acceptable not to eat because of the games. Games are not that important.

‣ They need to know how to control addictions. I want them to learn how to stop. I want to teach them to learn digital abuse.

‣ You get really fresh when you start playing games late, it wakes up, refreshes your brain. can’t sleep afterwards, terrible for the school next day.

‣ My computer does not control anything in this house. It is an old house, from the 70s. Changing electricity, and heating stuffs are really expensive, and you can never get it back, and you don’t get anything extra by selling the house later by those changes.

‣ Speaker ‣ Old rare one, Apple Hifi Speaker. It is so cool it

has a great sound, it still cost the same still I bought it like 7 or 8 years ago.

‣ I have an old apple remote control. ‣ It had physical buttons up here, but I never use

it. ‣ The volume is connected here, so i can do it

here. Good one, good sound. I kinda adjust the volume when I start the music.

‣ Remote control, for the parties or at the couch. ‣ The remote usually stays here. ‣ I have many parties, and it is really loud ‣ I still use iPod, 80 GB! Its like a time capsule. We

use it once in a year, every once a year, when we go to a summer house. it haven't been updated for 10 years- 12 years, 15 years?

‣ Sometimes i here music when I cook, sometimes, I listen to music from the radio, the crappy sounds makes me feel cozy.

‣ when I am not home alone,and cooking, music is not there. It is annoying to have a lot of different sounds.

2015

LIFESTYLE



‣ KITCHEN ‣ Microwave

‣ They are kinda disposable, they are so cheap.

‣ Just for the leftovers, to reheat. We don’t cook in it.

‣ Oven ‣ We use half of the times when we cook. ‣ I cook during the weekends, for two hours.

During the weekdays, It is usually Lisbeth who comes home earlier.

‣ We don’t use a timer. When it is done, it is done. you just check it by seeing the chicken.

‣ Preheat, it makes a sound. It is just like 5 minutes.

‣ I will be doing other tasks while preheating in the kitchen.

‣ The ventilation side is old, so it does not heating equally, but it is a bit of learning.

‣ Ceramic cooktop ‣ The plate indicates when it is hot, so you

shouldn’t put stuffs on it. ‣ It is not the best in the world, but it works? ‣ The product is a bit clumsy ‣ I am not really in to the advancement of a

kitchen stuffs, it is more about the taste of the food, the quality of the food

‣ If I would buy a really really fancy one, maybe the taste can get better.

‣ I like spending time here, so I don't care about saving 5 minutes.

‣ Watercooker ‣ We use it for tea and instant coffee. ‣ We are only dependent on it in the

weekends in the morning when we have less time pressure.

‣ If we don't have time for waiting filter coffee, we go for instant coffee.

‣ After 20:00, tea is an only option. ‣ Saturday, Sunday morning, we make coffee

for family time around the table. ‣ Radio

‣ Really terrible navigation, crappy. but it works.

‣ I don’t know how it got in the house. ‣ Its only me who interacts with me. ‣ If I need to buy one, I like it to be connected

with iPod with Bluetooth or something. ‣ I want it to be smart content-wise, but I like

it that it is a radio. ‣ It is really cheap, so getting it dirty does not

matter. ‣ Refrigerator

‣ The door works, as far as it works, it is no problem.

‣ Smoke alarm ‣ It goes automatic when there is a problem

(smoke). ‣ We actually bought an expensive one, we

have several of these which are connected, and one detects, the others should give an alarm as well. but it never worked, I lost my trust.

‣ The good(better) ones are really expensive. ‣ I would like my house to be more secured

and protected.


145


2015

LIFESTYLE



TWO PORCHES (TERRACES) / GARDEN One porch has the morning sunlight, and the other one has the evening sunlight. When the sun is there, it is warm and nice. ‣ Lighting ‣ I need to go out to interact. There is no light in

the garden, but only in the porches. the switches are with the doors.

‣ There is a light which turns on automatically by sensor.

ENTRANCE ROOMNot supposed to be shown to guests, we put dirty clothes here. When the kids are home messy, they can put out their clothes here without walking through the whole house. ‣ Washing machine

‣ we shower upstairs, and we just need to bring dirty clothes down here.

‣ I just use a couple of programs. ‣ It could be better, but I don’t even know

how all the clothes should be washed properly. Lisbeth is the super expert.

‣ I don’t know the meaning of the labels. ‣ I don't know when to go 900, instead of

1200, 1600… ‣ We have a charger here, buying new

batteries all the time should be expensive. BATHROOM ‣ Electronic toothbrush

‣ It has to be charged. BATHROOM (upstairs)

‣ No appliances in here.

BEDROOM ‣ No appliances in here. ‣ We read books at night, and turn it off when we

go to sleep. It turns off your brain. Reading for half an hour is to turn off the brains, but If you play with iPad or computer, it only makes your brain more fresh.

MANUCE’S ROOM ‣ iPad from the school ‣ For school work. ‣ They have security case because they need to

have one. ‣ Computer

‣ Manuce moved it next to his big brother’s room.

‣ Manuce moved a lot of stuffs to his brother’s room to play together.

CHRISTOPHER’S BEDROOM ‣ There is no smart, or any automated devices here. ‣ Computer ‣ TV

‣ It is with Chrome cast.

WORKSPACE (and also storage) ‣ Radio

‣ I have a friend who is a carpenter, and he said never to work without a radio.

‣ I like it, it is a messy, organized in a strange way, but I like it.

‣ The house, and the mountain bike needs a lot of maintenance.

STORAGE ‣ Freezer

‣ The only freezer in the house. ‣ We don’t have a space for a big integrated

fridges. ‣ I think putting vegetables in the fridge is

more important, putting a freezer in house will be a sacrifice for that.

‣ It will be a nightmare if they automate stuffs and if it does not work, they will not know what to do at all.

‣ Nightmare for fixing automated stuffs.

2015



BENEFICIAL SCENARIOS ‣ I am not really sure if it could be improved. ‣ I guess it would be difficult just as a remote

control. ‣ Because you will have to be in a certain space to

do a certain stuffs. ‣ Or if it is a voice control, if you listen to music… ‣ I guess in the future, I will like it. but I think it is

still very immature. ‣ I think it is a half solution. ‣ For me, gesture control has to do really smart for

me. like chopping up tomatoes, like a robot, really automated.

‣ If the difference is so small, it will be an expensive small change.

GESTURE CONTROL EXPERIENCE ‣ Kinect/Xbox ‣ I read about it, and know how it works, but I did

not get to use it. ‣ I expect it to be really good and accurate, from

the ratings that I have seen. ‣ Leapmotion ‣ I have heard of it, since I am sitting near the

technology guy in the company, and he has a lot of gadgets.

‣ Tasks that I do like thousand times in Photoshop, such as lining, if i have a gesture that could do it, if it could be saved as a short cut, to save time and make it faster and more efficient, I would DEFINITELY consider adopting it.

‣ If It could help me with accuracy and speed in my work.

‣ The shortcuts are working, but if it could be better, using my fingers, arms, necks.

‣ I would like to see designing with gestures, with brainwaves, more interface that is actually connected with your mentality/creativity.

‣ Designing is already connected with your personality, so why not extending it.

‣ I saw a video about it, and it looked smart, and thought it might be useful at some point.

‣ MYO band ‣ - ‣ Voice control ‣ -

SMART LIVING EXPERIENCE

ATTITUDE TOWARD SMART LIVING ‣ When you have kids, I cannot spend a lot of

money of gadgets and stuffs. ‣ It is about life quality, gadgets will get you more

disconnected with your family. ‣ I might not want any more devices, distractions

than we have already, such as Xbox, playstation, ‣ I like the voice control that it is disappearing. I

see ‘disappear-able’ as the future. ‣ Stuffs that you cannot see do something for you,

that is where we are going. ‣ I don't want an apple watch, google glass on me,

I think they are lame. ‣ They are just not built yet, because of the size,

but once the size gets smaller enough and disappear, I think that is where you will see some brilliant stuffs.

‣ With the technology but very subtle, not intrusive. ‣ I would hate to have a watch vibrating which

distracts me. ‣ While at work, I cannot focus when people poke

and mail me all the time, I turn everything off. ‣ I believe that not being distracted is very very

important. ‣ It is a part of smart living, not to embrace all the

distractions. ‣ Not to spend whole life on emails, it is not making

you smarter nor do creative stuffs. ‣ Thermostat ‣ Love it, i could save money, and more control

centrally. ‣ Thermostats are bit hard to control, not really sure

if i am doing it right always. ‣ I need more intelligent heating system. ‣ I don't wanna go change the temperature all the

time. just a task. just getting rid of one distraction. if i know what to do, and i would never see again.

‣ I hate technology that needs maintenance. ‣ you don't have to always update and do stuffs -

unless you want to do. ‣ Comfortable, but not increasing extra tasks. ‣ No emails. how many hours do you waste

checking and answering them? ‣ I basically value my stuffs, but I always consider

an upgrade or new purchase if it make me and my family more happy.

‣ For me smart has to add happiness.

146

2015

Respondent #08 Connie

It should add something in your life, with its extra features, which could be for practicality, or fun.“

Name Connie

Age 26 years old

Primary needs: For practicality or fun / Security Biggest Pain points: Difficult setup and usage

“

Nationality Danish

Living Arrangement Couple / Apartment




Household Appliances Intermediate


2015

LIFESTYLE



I live here as a couple with my boyfriend, non-married couple.

LIVINGROOM ‣ We spend our time the most in the living room,

watching TV and listening to music. ‣ TV

‣ I watch TV alone, because we prefer different channels.

‣ I sometimes approach the TV and control it with attached buttons only if I cannot find a remote control.

‣ - Unless I have to find a new channel. Finding out which button to interact is hard. I can’t remember from time to time, which one to press. in that case, I will control with the buttons on the TV.

‣ Using a remote control is okay otherwise, as far as it is found.

‣ Afterwork, just to relax, alone. I am home 2 hours earlier than my boyfriend. I watch for an hour.

‣ Speakers ‣ No idea how they are connected, my

boyfriend installed everything. ‣ If i need to control the music, i wouldn't

have anything in my hands. ‣ While cleaning, washing dishes, tidying up,

or other house chores. ‣ I use interface on the device.

‣ I have to be here to use the remote control anyways, once I am here, I can just press the buttons on the speakers.

‣ I would probably use my foot, to go to the next song, adjust volume if I were doing the dishes.

‣ Does not happen often, because I have a radio in the kitchen.

‣ Philips HUE ‣ One around the picture frame, and other

ambient lights in the room. ‣ Control with smart phones. ‣ One light is a quite bright incandescent

light, if it is too hot, I will turn it down. ‣ If i am tired, I prefer deemed light. ‣ I could control everything with my

smartphone, but I use switches to turn the lights on.

‣ If I turn the light on with my smartphone, it makes funny sound that makes me annoyed so I use the switches.

‣ We have been trying different apps because we did not want to pay for Philips app. then sometimes things does not work properly.

‣ Decorative Lamp ‣ It is a Mario lamp, inspired by the question

marked box in the game. ‣ It has different sounds, and trigger by

gesture. ‣ But it started turning on/off by itself,

making sounds, so we had to unplug it for now.

‣ I bought it for my boyfriend. ‣ I knew how it worked before purchasing. ‣ Before it got broken, it was fun to play with

it. ‣ Sensor was pretty accurate.


147


2015

LIFESTYLE



KITCHEN I spend a lot of time here, because we don’t have a dishwasher. I am usually the one who washes the dishes. I do cook, but I don’t enjoy. ‣ Ceramic Oven

‣ I do cook couple of times a week. ‣ It is straight forward to use. ‣ You just need to know, the heat will go

away gradually, unlike induction. But it is what I grew up with.

‣ If it is a dish that needs to sit in the stove for 2 hours, I would put the dish and come and go checking every 15 minutes.

‣ I watch a clock to check the time. ‣ We have two (analogue) timers on the

fridge, but we rarely use them. ‣ Its our habit from a student housing. when

we forgot to bring a smart phone. ‣ Fridge ‣ It has a light, and buttons for adjusting

temperature. ‣ Sometimes I need something from the fridge

while cooking, either I get it, or ask my boyfriend to get it.

‣ If i drop something, I will pick it up later, I don’t see it as a problem.

‣ Microwave oven ‣ I need the stairs to use it, it is too high (on

the fridge), it was the only place to put. ‣ We use it only for reheating. It is just nice to

reheat in a bowl. no extra pot or cooking.

‣ During the wintertime, we would use it more.

‣ Since now we have it, it feels annoying when we don’t have it.

‣ Radio ‣ I listen to the radio when I do the dishes. ‣ Sometimes when i cook, but then i am more busy

with stuffs. cooking and washing the dishes are not the same.

‣ I start doing the dishes first and turn on the radio in the middle.

‣ I do it with my wet hands. ‣ Often, I do the dishes with a brush, so my hand

with the brush is not very wet. ‣ But I would still turn it on with my wet hand if i

were doing dishes with a sponge. ‣ Kettle ‣ We use it for making hot water for cooking

pastas. ‣ 3-4 times a week. ‣ When I am waiting for the water to boil, I do other

things in the kitchen, when I were cooking. ‣ When hot water was for a tea, I would forget it

easily. then I have to do it again. ‣ It does not annoy me, if i really needed a tea, I

wouldn’t forget.

DINING ROOM / OFFICE ‣ Light ‣ The bulbs are all connected together. ‣ When we eat, we switch them on.

‣ When we have something for the dining room, we just turn it on.

‣ When your hands are full, you just use your elbow. ‣ Desk Lamp ‣ The lamp is with Philips HUE, but I rarely use that

function. ‣ Because it is hanging under the desk, I have to

locate it every time, but I don’t see it as a big problem.

‣ Tetris lamp ‣ I don’t control it, and It is not plugged. ‣ Because the switch is on the back of the block,

and it is hard to control it without messing up with the whole setup. so I just use it as an installation.

2015

LIFESTYLE



BEDROOM ‣ TV

‣ After dinner, before sleep. ‣ We control it with a remote. ‣ It is usually on my boyfriend’s side. I fall

asleep earlier, so he is the one who switches it off.

‣ Sometimes when I press A button, the TV turns off. some loose connections somewhere.

‣ Tablet ‣ I would just lie in bed, have it on my

stomach, and facebooking. ‣ when my boyfriend is watching the TV

something that I am not interested. ‣ Just software problems sometimes.

‣ Light ‣ This is our temporary night lamp. ‣ We don’t interact much.

‣ Wakeup lamp ‣ My boyfriend uses it. ‣ I can get up with a sound alarm instantly.

BATHROOM ‣ Washing machine

‣ We have two dirty cloth baskets, one in the bathroom for my boyfriend, and the other one in the bedroom for me.

‣ Mine is wood, should not get wet, and the other one is just plastic.

‣ I have no problem collecting all the laundries again, separating them into ikea bags.

148

2015



BENEFICIAL SCENARIOS ‣ Not really, but I can relate to the light. ‣ We have the light next to the TV, and if it is on,

then someone has to get out of the bed. ‣ It is annoying. ‣ I wouldn’t call it a fight, but we discuss who’s

responsible for turning it off by entering the room the last.

‣ It happens every night. ‣ When I go to bed I would ask him to.

GESTURE CONTROL EXPERIENCE ‣ Kinect/Xbox ‣ Leapmotion ‣ MYO band ‣ Voice Control ‣ ETC ‣ I used none of them.

SMART LIVING EXPERIENCE ‣ MEANING ‣ It is maybe about technologies, that can be

controlled in different ways, or that serves extra purposes.

‣ For example with the HUE light, regular bulb with white light, but it can also change its color

‣ so It is something extra. ‣ Could be fun, comfortable, but also practical. ‣ Especially, when you have disabilities. ‣ If the hangers that cannot reach could come

down to you… I would also call it smart, or smart living.

‣ It should add something in your life, with its extra features, which could be for practical, or fun.

‣ If it is about the light, it is just fun than practical.

ATTITUDE TOWARD SMART LIVING ‣ Technology ‣ I would accommodate new technology only

because it is fun. ‣ It has to be easy to set up, simple interface, and it

should work without major problems. ‣ Since we don't have many here, I don’t know if I

have a big problem. ‣ Usually its the set up that can be annoying, but if

you have to do it only one time, I can bare it. ‣ As long as the normal interface is understandable

and easy, its okay. ‣ I decide to buy or not, which i usually say no,

because it is expensive. ‣ Even though I could afford it, I ask to myself ‘do I

really need it? no, it is just for fun.’ ‣ I would save money. ‣ Philips HUE light was not necessary, I could live

perfectly without it. ‣ Fun purpose, I can have it, but also I don’t have to

have it. ‣ Most of the things are for my boyfriend. ‣ Smart living is not only for the necessity, but also

for fun. but people with constraints, it would be more about necessity, but I hope it to be fun.

‣ I like the ones that enable you to keep an eye on your house that you can see, and control lights, or video camera etc. I like those kind of technologies.

‣ It is for the necessity.

‣ I would have it, if i were not living in a renting place.

‣ We don't invest a lot of money in the house, because its not ours.

‣ I do want it when we are away for holidays, so we wouldn't figure out we are robbed only after we got back home.

2015


if it could potentially save money, make my life easier, and make the world better place, then it will definitely be interesting“

Name Anders

Age 26 years old

Primary needs: Comfortable, economical, sustainable Biggest Pain points: Learning more makes life more complicated

“

Nationality Danish

Living Arrangement Nuclear Family (1yr old son) / Multiple floor apt.

Smart Living Experience Intermediate



Household Appliances Medium



149


2015

LIFESTYLE



KITCHEN & DINING ROOM ‣ Probably this is the space where we have our

most of the appliances. ‣ We cook and eat, we also spend a lot of time

packing, repacking, unpacking for baby stuffs. ‣ Water cooker

‣ The only thing that we have out here all the time, is the water cooker and the coffee grinder. It is the water boiler that we use daily basis. for coffee, water, for sanitising baby stuffs, baby food.

‣ It is fairly easy. ‣ I hold Anker on one hand and operate the

device. ‣ Gas stove and electronic oven

‣ He can’t reach the stove top yet, so it is never a problem to use the gas.

‣ But if the oven gets hot, he can maybe get burnt. But we have been testing about how hot it gets, and the oven does not get too hot. So we are not worried too much.

‣ Anker walks around, and I will cook something.

‣ I don’t hold Anker and cook, I could use a baby carrier, but it is boring for Anker.

‣ Ventilation (range hood)

‣ Sonos speakers ‣ It is very easily controlled sound system. I

turn it on and off by just pressing the button, but I usually use with my phone.

‣ I can play all kinds of music from Spotify and radios from my radio app.

‣ I have my phone with me 80% of the time. ‣ Sonos app is easy, so simple. ‣ we have four speakers, and I named it after

the rooms: kitchen, bath room, office space, living room. I can choose how many speakers i want to play. I can also have two of the speakers to play one same song, and the others to play different songs.

‣ You accommodate, you wait to control Sonos when you have to babysit, and you dry your hands before using the app.

‣ I never have an experience that I absolutely need to change the music while I am doing the dishes. If I really need to, I will maybe just dry my fingers and then control my phone.

‣ It is not an essential tasks. I never thought of it as a problem.

‣ But of course, if you really have to, It will be a challenge…

‣ Refrigerator ‣ It is really fairly easy. It has this

thermometer screen, and it has buttons to control. if open for too There was a time when I had to keep an eye on the baby all the time, it was the biggest challenge all the time.

‣ Back then, I wouldn't cook, I wouldn't do other stuffs when I was baby sitting. that was like… when he just started to crawl, about two months, between 6-8 months.

‣ Lighting system (Lamp) ‣ The dimmer is not with the switches, but

attached to the ceiling, and I would really like it to be with the switches. because this place is a rental, we cannot really do it.

‣ I can reach it, but my girlfriend can’t. ‣ But we never actually change, so we are

still pretty happy. if too dark, we can just turn on more lights.

‣ We don’t have Philips HUE, actually because I don’t like the light. I don’t have anything against artificial light, but it does not fit to our style, and the colors that HUE gives is too artificial.

‣ It is more gimmicky. its not what you want all the time.

2015

LIFESTYLE



LIVINGROOM This is a space where we spent our time the second most. ‣ Baby gates (attached to the staircase entrances)

‣ I would LOVE it if it were electronic. ‣ You could just turn on a switch. ‣ It needs some finish in order to click it in. It

is not hard, but it needs some attention to close it properly.

‣ Smart TV ‣ We spend a lot of time watching the TV, not

everyday, maybe like 5 days a week. ‣ Anker is not really into TV yet. If it catches

his eyes, he thinks it is fun to watch for 2 minutes or something.

‣ Once the baby is into bed, we stay home and watch TV, partly because we have to take care of the sleeping baby, but also because we are too tired.

‣ Ever since we had Anker, we spend more time with TV.

‣ We misplace the remote control often, but it is rare that it is hard to find in a short time.

‣ Using a remote control the only way that i know how to control the TV. I don’t even know if there are buttons on the TV.

‣ It works perfectly, as long as you know which button does what.

‣ I had to get used to all the functions it in the beginning, it took me half a day.

‣ I remember one time, I wanted to program these (colored) buttons, I searched manuals online, read them through, and I was still not able to program.

‣ Long loading time of the content(channel) list annoys me.

‣ LP player ‣ Traditional one, connected with not strong

enough amplifier. ‣ I rarely use it, but i think it is cool. It is now

rare for use to sit down and put on a LP. ‣ The whole set up(player, CDs, amplifier,

speakers) is just for this system. normally we use Sonos.

‣ If Anker grows up a bit more, we will use it more.

OFFICE ‣ Baby monitor

‣ we put one in the baby bed, and we carry another one.

‣ It is definitely helpful. ‣ We put Anker in the yard when he sleeps,

and we do other stuffs, it is really nice that we can utilise our time better, instead of sitting next to him all the time.

‣ Computers ‣ We don’t have any computer usage, just for

checking emails, government mails, paying bills.

BATHROOM

‣ It used to be bathroom, but we made a diaper station for Anker.

‣ Washing machine + dryer ‣ It is all-in-one, you just choose a program

and then it is a full service. ‣ It is always easy. ‣ We really like that it has a child-lock

function. ‣ One annoying thing is that On/Off function

still works when child-lock function is on, so Anker can ruin the whole process by turning the machine off.

‣ Blinks and noises from the machine is so fun(hilarious) for Anker.

‣ There is no danger for Anker.

‣ BEDROOM 1 ‣ We don't have any electronic appliances. ‣ We don’t even have lights. We like to wake

up with natural light. ‣ Anker’s bed. which he sleeps in only during

the night. ‣ for a nap in the daytime, we put him in a

stroller in the yard.

‣ BEDROOM 2 ‣ Babyroom in the future. ‣ Playroom/Guestroom for now) ‣ We only have the lights. ‣ We have a plan to build a home theater..

150

2015



BENEFICIAL SCENARIOS ‣ I think what gesture control devices could achieve

the most is to remove a bit of annoyance. I don’t think it could make a big difference.

‣ You can always adjust your daily routine to the circumstances depending on the needs that a little boy (Anker) has.

‣ It is easier to make those adjustments (for example, to wait until cooking is done) than buying, and learning a whole lot of devices to solve it.

‣ It is complicated to invest the time in it to figure out how it works.

‣ The cost, it is expensive to buy a lot of stuffs. ‣ The major thing that keeps me from buying all the

time is the awareness of that I need to learn it, I try to limit how much i need to learn as much as possible.

‣ I am trying to keep my life simple.

GESTURE CONTROL EXPERIENCE ‣ Kinect/Xbox ‣ I used it only very shortly, and that was a lot of

fun. ‣ I like boardgames and cards, never played

computer or consoles. ‣ Things like Kinect/Xbox, or Wii, they really speak

to me. It is not as passive as sitting with a keyboard and a mouse or a playstation controller. That activity, and moving around with your body makes a lot of fun for me.

‣ I am not against it. ‣ More physical gaming experience.

SMART LIVING EXPERIENCE ‣ But once in a while, Sonos sound system, I really

wanted to get a music system which i can control with my phone, because it annoyed me that I need to go somewhere. Also to have a set of speakers that could talk to each other in an apartment of two floors meant a lot of wiring. I wanted wireless and control with my phone. I did a LOT of research, what is the right kind of speakers for me.

‣ What I especially wanted was very user friendly. Sonos promotes themselves as very user friendly, very easy to set up, very easy to control (app), it is true. it is very easy, it is a little bit more expensive than other bluetooth speakers.

‣ I was skeptical at first. ‣ Absolutely met my expectation. I can recommend

the product. ‣ Sometimes, smart product surprises me, learn to

like something that I did not expect. ‣ The smart TV, surprised me with its functions. We

bought it because we cannot buy any quality TV without a smart function. and now we have Netflix that we really like to use.

‣ Netflix, it surprises me that I liked the TV to have those kind of functions. Browsing different contents, it is really easy. we push a button on a remote control and Netflix is on the TV.

‣ We used Netflix before, we linked the old TV with HDMI cable, which was a barrier and we rarely used it.

ATTITUDE TOWARD SMART LIVING ‣ I would like to have something like NEST. ‣ During winter, going around adjusting windows

and thermostats is just annoying. If I can have like intelligent heating/ventilation system at home, it would be really nice.

‣ Velux ventilation - danish window company, which controls indoor atmosphere using heat, light, ventilation - the most advanced. they have cool products.

‣ For Indoor climate and energy efficiency. ‣ It takes a lot of time before it actually brings you

an economical benefits. ‣ Danfoss. Intelligent thermostat, which recognises

window openings, falling temperature, and controls heating system.

‣ NEST, and Velux system do everything. ‣ Comfort, economical and sustainable benefit. If I

can see more than one smart thing, if it could potentially save money, make my life easier, and make the world better place, then it will definitely be interesting.

‣ Turning on Sonos by clicking my fingers, would be a very minor improvement, or no improvement.


151


D.IDEATION 1: ITERATIVE PROCESS

1. SECONDARY RESEARCH

Home automation domain

After failing in gaining many useful insights from a previ-ous stage, home automation concept was explored. As the concept has a lot of automation features, Home automation was considered very relevant. Use cases were built refer-ring to well-known home automation categories (Wikipe-dia). Among various tasks that home automation can serve, HVAC (heating, ventilation, and air conditioning), Lighting, Shading, Security, Audio-visual, intercom, etc., Heating, Lighting, Shading, Security were chosen to be good examples to demonstrate the concept, as it more relevant to a family experience. The other categories such as Audio-visual, In-tercoms, Domotics were excluded, considering the charac-teristic of the main target group, who are less focused on entertainment, and digitalized and automated gadgets. The cases built up are shown below.

#1 Waking-up triggering devices to start

152

APPENDIX D.

#3 Leaving house securing home

#2 Shivering (o.g. fanning) gesture affecting heater

IDEATION 1: ITERATIVE PROCESS

153


Figure. Persona created to identify user needs during the iterative process

HOUSE

LIVING ARRANGEMENT

TECHNOLOGY ACCEPTANCE

EXPERIENCE & LEARNABILITY

VALUES

NEEDS

A flat with multiple floors

Mark (28) Father working at a company Rose (30) Mom on maternity leaveOllie (0) 3 months old infant

Conservative

Novice and Easy user

Closely watching the infant growing up day by day with less hazard will be make a grateful moment.

Busy family will appreciate small peaceful moments with the kids in the morning.

All the of family members have their daily routines. As there will be less and less time to be together, morning time when they are getting together is precious.

Taking care of an infant is demanding.Extra caution and effort is needed for babysitting.Potentially harzardous situations for the infant should be avoided.Multitasking: Keeping an eye on the infant while dealing with other daily tasksThere are a lot of baby products at home.

HOUSE

LIVING ARRANGEMENT



VALUES

NEEDS

A house with multiple floors in a city

David (37) Father working at a company Sophia (35) Mom on maternity leaveEwan (4) PreschoolerTim (1) 1 year old toddler

Conservative


Very hectic morning, with two very energetic kids. The toddler always needs to be attended, a brief moment of taking eyes off can cause small and big problems. The older one occasionally needs some help of the parents.Toddler, and sometimes the preschool one can get so excited about nothing - blinking lights.There are tremendous amount of house chores and tasks to be done, also while dealing with both of the kids.

HOUSE

LIVING ARRANGEMENT



VALUES

NEEDS

A house with multiple floors in a surburb

Chris (28) Father working at a company Holly (30) Mom on maternity leaveHannah (10) First childJoe (6) Second child

Conservative


Gradeschoolers and parents can now have some real socializing experience with each other at home.However they have their hectic moments in the morning, since everyone has to be prepared for their day and there are some tasks to be done as well. These gradeschoolers sometimes can help the parents with various household tasks.

FAMILY #1 PARENTS WITH AN INFANT

FAMILY #2PARENTS WITH A TODDLER & A PRESCHOOLER

FAMILY #3 PARENTS WITH GRADESCHOOLERS

154

Product Usage CycleA product-usage scenario was formulated to inspired the future design process of the concept.

A DIY smart home product usage cycle (Woo & Lim, 2015 ) was used as a basis then revised to be more suitable for the target user of this project whose technological acceptance and Smart Living experience are fairly low. The low technol-ogy acceptance and Smart Living experience were consid-ered to have influence especially on the early product usage cycle, when the users have to put effort in understanding and learning the device. When compared to the DIY smart home product usage cycle, significant changes and major insights can be found in the Table below.

Table. A DIY Smart Home product usage cycle and design implications

At first, the framework for the usage cycle was constructed as seen below.

Initial Installation -> Implementation -> Routinization -> Actual Usage -> Customization

Thereafter, a more descriptive storyboard was created to communicate the initial concept.

2. INITIAL CONCEPT

Usage scenarioInitial InstallationEssentially, the device has to be connected to internet.

ImplementationImportant settings have to be made. Information such as user profile, list of spaces and products should be put in the system by the user. The setting should be minimal, and as easy as possible to follow. This is done by using website, or web application, since use of the computer seemed the most logical for dealing with a rather complex tasks that requires various inputs from user side, as mentioned before.

RoutinizationOnce implemented, which means initial settings has been made by a user, the system starts to track and learn users’ life patterns. For example, when and where the user wakes up, and what kind of products that the user use. Based on time, location, user, and gesture, the system will set automation rules based on its learning.

UsageAfter the system setting rules, users can browse what has been identified and set as a pattern by the system, on the same web platform used in Routinization. From this point, users can make use of the intelligent the system without any hassle.

CustomizationThe system allows the users to fix some rules depending on their needs, in case the users’ life pattern has been changed or they simply did not like any automated events set up from their patterns.

FeaturesThe basic idea for the concept is a A system that detects natural gestures and translates them into relevant tasks and performs the tasks automatically.

Easy step-by-step settingsThe settings are done on computer, Web/desktop application. Guides user to follow a step at a time with comprehensive image based explanation. Most importantly, the guide relieves a burdening process, thinking of how to implement a smart system. Instead of asking plans, the system only asks about the facts. Users only have to provide information about actual spaces and products used, instead of coming up with various usage scenarios by them.

LearnabilityLearnability directly corresponds with the intelligence of the system. The system recognizes gesture - task rituals, and it is able to read the context and perform tasks automatically.

Use of a camera sensorTo detect gestures, a camera-based sensor is used. It has to be located in a space where the user would like to interact with.

Master ControlMaster control is a feature that enables users to stop any unwanted automated tasks, or start any deviation.

Activation gesture for Master ControlGesture recognition system being always activated does not make sense. Firstly, people uses a lot of intended and unintended gestures, and it can cause a lot of confusion while interacting with such a system. Secondly, camera based gesture recognition that are always activated to track complex gestures might be an unnecessary energy consumption. It would be better if the system can be activated when it is needed.

Scalable Control (giving control to kids)When kids are registered as users, their gesture can basically trigger tasks. Furthermore, with Master Control feature, they can also experience the interaction with products at home, which will include the kids as users of the house, thus involving them more in household tasks as a part of being a family.

APPENDIX D. IDEATION 1: ITERATIVE PROCESS

155


156


157


158

3. USER RESEARCH 1: CONCEPT IDEA EVALUATION

OBJECTIVESThe objective of this first user test is to collect various opinions of the initial concept. The research question here was to figure out how it feels like to have an intelligent system triggered by gesture in general. As such, the user test covered diverse topics regarding the user experience of the whole usage cycle, and different features of the concept.

METHODSParticipantThree participants who fitted in the value-seeking hesitator criteria were recruited again to give their feedback.

ProcedureA video prototype was created to introduce the concept to the participants. A video prototype was selected as a medium to communicate the concept effectively for this stage, because the concept has a complicated interaction scenario with various use cycles and platform. The video was created based on a storyboard according to a timeline, presenting several use cases at the end. Introducing the initial concept in such a way was believed to depict how the concept work clearly. Considering the given amount of time for the project, the interviews were conducted in a form of an online survey. The video prototype was attached along with the survey. Each participant was presented with a video prototype first, then be asked to reflect the opinion on the concept. The contents

of the survey included questions related to; the general idea of the concept, the settings, learnability of the system, use of a camera (instead of a wearable), master control, and scalability. See the figures below for more details. The evaluation criteria included the easy of understanding and use, intuitiveness, expected benefits and concerns of the system.

RESULTS AND DISCUSSIONRegarding the basic idea of the concept, participants appreciated the feature that understands users’ life patterns by detecting users actions. The participants considered such a feature to correspond with intelligence, as the future of Smart living. Simply mapping gesture and function would not contribute much to Smart living was mentioned again.Participants were concerned about having one unified system that only uses gestures, as it could go wrong and mess up with life very bad, when the system is underdeveloped and gestures are misinterpreted. Many of the feedback concerned the Implementation part. It is thought that the implementation part, which included various UI designs, was more clearly visible in the video.

Initial settingAfter the initial setting, users should be able to experience the power of control right away, instead of waiting a week for the routine to be set up. So, being able to set a master control gesture in the initial

RQ1 The conceptIn this concept, users are not required to make any abstract and peculiar gestures to trigger tasks to be done. Instead, their actions - purposeful gestures that has specific meanings and in their living environment will serve as a trigger.

The example of purposeful gestures are as follows:- Sitting up and standing up to wake up from a bed in the morning- Shivering from feeling cold- Walking, or running to move to one place to another

In general, how do you feel about using your ‘purposeful and contextual’ gestures to control your environment, instead of performing abstract gestures?

Does it feel more natural to you?Unnatural - Natural

Does it feel easier for you to understand, and use/interact with the system?Difficult - Easy

What to you think as the biggest benefits for using such a system?

Do you have any concerns related to your topic?

RQ2 The SettingsWhat do you think about the initial settings part?In the settings part, the system provides a step-by-step guide to set up the whole system. You don’t have to read manuals, but just follow what the system says, registering your products one by one.

How does it feel to follow the guide in general? Very difficult - Very easy


RQ3 System LearnabilityInstead of you thinking about your daily living and setting complicated automation rules,the system will read your patterns and set, and suggest automation rules for you.

What do you think about the intelligent system that learns your daily patterns, and makes suggestions for you?

What do you think about the screens that shows rules to you, how easy is it to understand Difficult - Easy

and what is the reason?

What would you prefer an image based description, or a text based description? What feels more easy and friendly to you when comparing the rule palette and the setting screen from another application (Samsung SmartThings)? Please refer to the images below.

ule alette a su art i s



Do you have any recommendations related to your topic?

RQ4 Use of a CameraThe system uses a camera sensor to recognize users and their gestures. This will remove the chance of using wearables and various (input) sensors.

Would you prefer to use a camera based system than using a wearable?es uld refer a ca era ased syste

uld refer a eara le


159


stage seems to create better experience.

Master controlMaster control is a definitely required feature. At a glance, setting a specific master control gesture does not correspond with the main theme of the concept, which is to be as natural as possible and integrated in users existing life patterns. Also, having a sense of controlling is not necessary for a smart system, regarding non-techsavvy users. However, this master control feature is very important as a safety measure.

Activation ‘signature’ gesture for Master controlWhen it comes to signature gesture to activate Master control, positive opinions were collected as well. Although users did not seem to be into customization or empowerment from the previous interview, it seems like that users are willing to accept the feature as the activation gesture is functionally necessary. When it comes to social awkwardness, being at own home environment with family may relieve the pressure to some extent. Also, to decrease awkwardness, the master gesture control can be set subtle to minimize the discomfort as well. Furthermore, setting master control can offer a good opportunity to involve young users (kids), providing empowerment and some fun. In that case, the system allows young users to set very distinctive gesture to experience the magic of controlling, in distance.

Use of wearable vs. CameraWearable was not preferred in general, except for one participant who did not want any camera in his house at all. For the rest of the participants, it does not have any emotional benefits on the users. Being used in a home environment makes it even worse, as home is considered where people can take a real rest, and wearing an extra device does not fit into this context at all. (However, when it comes to gesture recognition, use wearable may have better performance for little effort). While using a camera based sensing seemed more preferred than using a wearable, there were some issues raised by the participants. The first issue was the surveillance. Some participant mentioned that they would not want to place cameras everywhere in their living space. Having cameras that ‘watch’ their behavior did not seem to make them feel natural and comfortable at home. The fact that video based data captured by the system to detect gestures could be hacked seemed concerning as well. When it comes to camera, the view of camera was also considered to be problematic. There is a chance that a user can block the camera or turn his back to a camera, in both of the cases the camera might not be able to detect gestures properly.

In either way of using camera, or wearable, it seems that the more invisible and natural, the better the system will be.



RQ5 Master ControlThe system has a master control feature, which activates controlling over distance, so you can start/stop any automated events whenever you want. This requires you to perform some abstract gestures to be activated.

How necessary do you think this ‘master control’ feature is?Very Unnecessary - Very necessary

What do you feel about setting your own special ‘signature’ gesture for this feature, that can be differentiated from other family members?



RQ6 ScalabilityScalability of control, allows you to give some controls to child(ren) so they can interact with surroundings as well.

Would you include your child(ren) to have a bit of control? Yes - No

Why, or why not giving a control to your child(ren), could you give some reasons?

What kind of experience do you think your child(ren) will have? Do you think your child(ren) will have fun with it?

160

Section 1 of 9

After section 1

Section 2 of 9

Gesture-controlled Smart LivingThank you for participating in this research. The objective of this survey is to gain insights in the field of Smart Living, regarding a new Smart Living system that utilizes a new way of interaction: Gestures. As a potential user, your input will be very valuable. Please spare a bit of your time and tell us what you think.


The video below describes a new system that enables Smart Living with a new interaction vision. Please watch the video, and tell us about your opinion.

Untitled Video


After section 2

RQ1 THE CONCEPTIn this concept, users are not required to make any abstract and peculiar gestures to trigger tasks to be done. Instead, their actions - purposeful gestures that has specific meanings and in their living environment will serve as a trigger.

The example of purposeful gestures are as follows: - Sitting up and standing up to wake up from a bed in the morning - Shivering from feeling cold

In general, how do you feel about the concept, using your ‘purposeful and contextual’ gestures to control your environment, instead of performing abstract gestures?

Long answer text

Unnatural

1 2 3 4 5

Natural

Does it feel more natural to you?

Very difficult

1 2 3 4 5

Very easy

Does it feel easier for you to understand, and use/interact with the

What do you think is the biggest benefits for using such a system?

Long answer text

Do you have any concerns related to this topic?

Long answer text


Section 3 of 9

RQ2 THE SETTINGSWhat do you think about the initial settings part? In the settings part, the system provides a step-by-step guide to set up the whole system. You don’t have to read manuals, but just follow what the system says, registering your products one by one.

Image title

Very difficult

1 2 3 4 5

Very easy

How does it feel to follow the guide in general?

After section 3

Section 4 of 9


Long answer text


RQ3 LEARNABILITY OF THE SYSTEM Instead of you thinking about your daily living and setting complicated automation rules, the system will read your patterns and set, and suggest automation rules for you.

Image title

What do you think about the intelligent system that learns your daily patterns, and makes suggestions for you?

Long answer text

Very difficult

1 2 3 4 5

Very easy

What do you think about the screens that shows rules to you, how easy is it to understand ?


161


Very difficult Very easy


Long answer text

What would you prefer an image based description, or a text based description? What feels more easy and friendly to you when comparing the rule palette and the setting screen from another applications? Please refer to the images below.

Image based description

Text based description

Other...

Image title


Long answer text


Long answer text

After section 4

Section 5 of 9

Do you have any recommendations related to this topic?

Long answer text


RQ4 USE OF A CAMERAThe system uses a camera sensor to recognize users and their gestures. This will remove the chance of using wearables and various (input) sensors.

Image title

Would you prefer to use a camera based system than using a wearable?

Yes, I would prefer a camera based system.

No, I would prefer a wearable.

After section 5

Section 6 of 9


Long answer text


Long answer text


Long answer text


RQ5 MASTER CONTROLThe system has a master control feature, which activates controlling over distance, so you can start/stop any automated events whenever you want. This requires you to perform some abstract gestures to be activated.

Image title

After section 6

Section 7 of 9

Very Unnecessary

1 2 3 4 5

Very necessary

How necessary do you think this ‘master control’ feature is?

What do you feel about setting your own special ‘signature’ gesture for this feature, that can be differentiated from other family members?

Long answer text


Long answer text


Long answer text


RQ6 ScalabilityScalability of control, allows you to give some controls to child(ren) so they can interact with surroundings as well.

Image title

162

After section 7

Section 8 of 9

Would you include your child(ren) to have a bit of control?

Yes

No

Why, or why not giving a control to your child(ren), could you give some reasons?

Long answer text

What kind of experience do you think your child(ren) will have? Do you think your child(ren) will have fun with it?

Long answer text


ALMOST THERE!Description (optional)

Very much notlikely

1 2 3 4 5

Very likely to buy

In general, how do you feel about using the system for your own apartment? Would you be willing to buy it, if the price does not matter?

Could you give some reasons behind your argument?

After section 8

Section 9 of 9

Could you give some reasons behind your argument?

Long answer text

If you could make any changes, what would it be?

Long answer text


Thank you!This is the end of the survey. Your input will be very helpful for this design research. We appreciate again for your time and effort.

Please leave your name here, so we know who you are :)

Short answer text

Please leave your email address.

Short answer text

5 responses

Accepting responses


RQ1 THE CONCEPT

In general, how do you feel about the concept, using your‘purposeful and contextual’ gestures to control your environment,instead of performing abstract gestures?(5 responses)

SG

In general i don't think it would make that much of a difference if a system could recognizegestures. but a system that could understand context is genius. The first case in the video is notreally relevant for me, or something i would invest in. But the second and especially the third isreally clever.

It seems like a good idea. It makes sense to adapt natural behavior.

I feel like I already know how to use the system and get started with the experience immediately. Itmakes sense that my natural actions and nonverbal communication can be translated intoaffecting solutions.

think I might trigger some actions without thinking about it - and thereby risk loosing control a bitover what is happing. It might be on purpose the heat is set low or that me sitting up in the bed,not necessarily means that my husband are planning to get up – this will vary each morning...

Does it feel more natural to you? (5 responses)

Unnatural 1 2 3 4 5 Nat0

1

2

3

0 (0%)0 (0%)

2 (40%)2 (40%)

1 (20%)1 (20%) 1 (20%)1 (20%) 1 (20%)1 (20%)

Does it feel easier for you to understand, and use/interact with thesystem?(5 responses)

What do you think is the biggest benefits for using such a system?(5 responses)

It could automatically remind me of and adjust some of my bad habits, like forgetting to turnsomething off. It would save me money and resources.

Benefits of saving money on heating. The kids would be more in the loop of controlling the homeright.

Not having to learn anything before I can get started. I don't necessarily have to be conscious ofmy actions .

For example light controls could be nice - so the light follows you around in the house, but I do notlike the idea that only one room is lit all night, because only one person is at home. It should fit thetask the light has e.g. work light in the washing room could be part of such a system - but not inthe living room. Think that the kitchen also has some obvious opportunities - to proved hands-free control ofturning the water on and off – or the oven or stove.

XCV

Do you have any concerns related to this topic? (5 responses)

Maybe I would drink more coffee, meaning to much.

The system could misunderstand gestures and preform commands that was not intended + in is aunnatural/under-developed way of interacting with products.

It needs to be adaptable to the nuances of the individual. There are universal gestures ormovements but, there are also anomalies that have to be taken into account.

Very difficult 1 2 3 4 5 Very0

1

2

3

4

0 (0%)0 (0%)

2 (40%)2 (40%)

0 (0%)0 (0%)

3 (60%)3 (60%)

0 (0%)0 (0%)


163


a lot - but it may be because I need the see the relevant context it would make sense to me first.The scenario about the child waking up is not realistic to me. Either they will call out - and I willwake up right away – or they will come in and crawl into my bed to cuddle up. I do not need eitheran alarm or the blinds to be rolled up. Perhaps we want to just stay in bed a few minute more :-)

XCV

RQ2 THE SETTINGS

How does it feel to follow the guide in general? (5 responses)


I can't vision a world were every product was compatible with one controlling system

ETR

Think you skip the part about how to install your appliances – also the wording is quiet technical.Think you could optimise the experience more by added more than just a set-up guide. Perhapsyou can also choose which gestures you prefer for each action. Also it seems like the features arevery default - perhaps you can as a user choose between different pre-set until the system gets toknow you better.

It seems that a lot of these screens can be replaced by interacting in the physical space where I'mwhere I'm attaching the actuators. Why do I have to pick a card that represents where I wasphysically. This part feels like an old way of doing things.

No

RQ3 LEARNABILITY OF THE SYSTEM


1

2

3

4

0 (0%)0 (0%) 0 (0%)0 (0%)

3 (60%)3 (60%)

2 (40%)2 (40%)

0 (0%)0 (0%)

What do you think about the intelligent system that learns yourdaily patterns, and makes suggestions for you?(5 responses)

This is my expectation for most future products.

Could be a nice feature - but concerned that there might be a lot of small annoying "mix-action"before you get to a fully optimised system. It will be like starting in a beta version. Also personalpreferences will vary a lot from person to person in a family – so concerned that the system willalso just reflect the common denominator in the family and thereby never satisfy the individual.Also there are some dynamics in a family I would be afraid to miss out. Why I turn on the light inthe living room to make it cosy for my kids - which my husband does not think about – but then hehas some other habits they will learn from. One example is that the faucet in the kindergarten atone point was change the a gesture controlled (for better hygiene). But then my youngest girlbegan to forget to turn off the faucet at home. Not sure, if it is because we should just change allfaucets to be gesture controlled - or if she should get at better feed back form the automaticfaucet that it is turned off.

That is a really smart thing, just like NEST slowly starts to learn your preferences. That is the futureog home technology, i am sure.

I think that the system is getting smarter over time is a must.

XCVX

What do you think about the screens that shows rules to you, howeasy is it to understand ?(5 responses)

and what is the reason? (5 responses)

Long lists are cumbersome. There is no context other than the headers so, it instantly makes menot want to look at it.

It looks easy when i see this picture. but some gestures will probably be more difficult.


1

2

3

4

0 (0%)0 (0%)

1 (20%)1 (20%)

3 (60%)3 (60%)

1 (20%)1 (20%)

0 (0%)0 (0%)

AGSDF

did not pay attention to this.

They are understandable, but maybe a bit to many options.

What would you prefer an image based description, or a text baseddescription? What feels more easy and friendly to you whencomparing the rule palette and the setting screen from anotherapplications? Please refer to the images below.(5 responses)


people use different input to understand the information

Speed, you dont want to spent to much time in this.

XCGGDFG

Images are easier to understand, but also easier to misunderstand.

I want to see and understand instead of see, decode, and hope I get it right.


not sure how long time you would keep coming back to this to optimise. But especially with kids inthe house, your habits change all the time as they grow up. Unsure if the system can keep track ofthis.

Seems tedious and I don't really understand what it is supposed to provide in terms of value. Myguess is that it's giving me suggestions for optimization.

0 0.5 1 1.5 2 2.5

Other

Text based d…

Image based…

0 (0%)0 (0%)

ERTERT

I think there are other areas that are more challenging.

No

Do you have any recommendations related to this topic? (5 responses)

think you should consider to show it instead of explaining it.

Use another form of presentation other than a list. Also, use mechanisms that allow me to seewhat's the most important element first.

No other than, do a lot of user testing.

ETETER

intuition is god. but you already know that.

RQ4 USE OF A CAMERA

Would you prefer to use a camera based system than using awearable?(5 responses)


Not sure what I would prefer. The technology that solve the task best I would assume.

I would forget the wearable.

HFGFG

0 0.5 1 1.5 2 2.5 3 3.5

No, I would p…

Yes, I would…

2 (40%)2 (40%)

164

HFGFG

With a wearable the system is dependent on it being charged and worn by the user. In a homesetting, you usually want to shed not only your clothes but also your tech.

You would have to have a lot of cameras to cover the use. Or else you need to be a certain place tointeract.


I do not like the idea of have cameras all over the house.

The idea of a camera on all the time freaks me out.

What happens when the camera's view is blocked or my back is turned.

FGFGT

surveillance. it must be a totally safe grid if a were to put up cameras in my home


Actually I would prefer to do stuff on my phone, computer or iPad, rather than a system withcameras.

Maybe a ceiling mounted camera could be a more omnipotent setup.

make them a un-noticable as possible.

RGRGR

make it as discrete as possible.

RQ5 MASTER CONTROL

How necessary do you think this ‘master control’ feature is?(5 responses)

1

2

3

4

1 (20%)1 (20%) 1 (20%)1 (20%)

3 (60%)3 (60%)

What do you feel about setting your own special ‘signature’gesture for this feature, that can be differentiated from otherfamily members?(5 responses)

Not really sure, but it seems a good idea to have a gesture to recognize each person.

As long as I'm the one creating it, I feel comfortable.

I think it is great. but i also think that it would be very easy to copy it.

OK I thnik - depending on how many gestures you need.

DFGDGD


You would perhaps quickly look like a jumping jack. Perhaps the gestures are easy to copy be others.

no not really

Not really...just what happens if I forget?

FGH

No


The system should let me know if my signature gesture is already used, or can be interpretedclosely to other know gestures. That would reinforce the sense of intelligence.

make the gestures is simple as possible, and do not let the gestures be mixed but with each other.

FGH

no

No

.. 1 2 3 4 5 ..0

1

0 (0%)0 (0%) 0 (0%)0 (0%)

RQ6 Scalability

Would you include your child(ren) to have a bit of control? (5 responses)

Why, or why not giving a control to your child(ren), could you givesome reasons?(5 responses)

DFGFTH

They live here too. They use all my technology (smartphone, tablet, etc.) so, why wouldn't they alsointeract with their home.

When you are a family in a house, everybody is important. So I think that children should havealmost full access.

I would also teach my kid to turn the heat on and off.

I would think that they should feel like it is their home as well - further it is a big part of growing up,that they are able ot perform tasks on their own e.g. get a glass of water them selves, turn on thelight etc.

What kind of experience do you think your child(ren) will have? Doyou think your child(ren) will have fun with it?(5 responses)

They may have fun activating/deactivating depending on how visible the function but like mostthings at their age, they'll soon take it for granted and treat it like a utility.

I guess it will be an integrated part of their everyday life.

Not sure with fun. But the would for sure love to skip all the boring tasks but automation.

Yes

No

100%

I think it will be fun for about 30 minuts. then they will forget and get used to it.

FGHRY

ALMOST THERE!

In general, how do you feel about using the system for your ownapartment? Would you be willing to buy it, if the price does notmatter?(4 responses)

Could you give some reasons behind your argument? (4 responses)

I hate mundane tasks. The less I have to consider them the better.

I think the selling point should be more defined and a would have to see a stronger benefit if ashould buy it.

I do not have that much trouble by the tasks I have to perform in my home besides hovering andsorting the washing.

I think it might be more relevant to people with special needs or disabilities.

I would look in to it in the future. But atm im in an older house (from 1970) and the cost ofupgrading for example the heating system would be to expensive. But I would be interested inparts of the system.

If you could make any changes, what would it be? (4 responses)

I think we are very attached to our phones and that they should be included in the system.

.. 1 2 3 4 5 ..0

1

2

3

0 (0%)0 (0%)

2 (50%)2 (50%)

1 (25%)1 (25%)

0 (0%)0 (0%)

1 (25%)1 (25%)


165


make it a systemt that i don't need to interact with, but a system that recognizes context and actaccordingly to that instead,

The video that is captured should be stylised or degraded so I don't feel I'm being watched ormisused.

Focus on a few specific tasks the system can perform and then make sure this is relevant the user– instead of letting the technology control what the system can do.

Thank you!

Please leave your name here, so we know who you are :) (5 responses)

Tommy Jensen

Damoion

camilla waldstrom

Anders Tolstrup

HELLO HELLO

Please leave your email address. (5 responses)

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

166

4. INTERIM PROTOTYPE BUILDING

An interim prototype was built, using Kinect and Processing. It mainly aimed to build a system that can detect shivering, and fanning, to test that thermostat scenario. Using the Kinect’s ability to detect limbs and joints, the system was made it possible to detect the desired movements. However, it was decided not to use in the future user tests. The reasons were: the system required the participant to stand in front of Kinect camera and perform the gestures within the reading range, the system had a bit of lagging time to detect gestures and process them, and the gesture detection based on limbs and joints were somewhat limited to detect some subtle gestures as well.

For the flexibility and the speed of gesture detection, the Wizard-of-Oz method is considered positive.


Figure. The use of Kinect camera and Processing for building a working-prototype to detect shivering/fanning gestures

167


168

5. USER RESEARCH 2: APPLICATION OF GESTURES

OBJECTIVESThe objective of this first test is to collect various opinions on the concept, regarding the whole usage cycle of the concept (product). This stage was considered necessary since the testing with the video prototyping did not yield many feedback regarding the actual usage, using gestures to control the environment. The research question was to figure out naturalness (as opposed to awkwardness), intuitiveness of the system manipulated by gesture.

METHODSParticipantTwo participants, one in 20s, and the other in 30s, took a part in this research.

ProcedureThe main focus of the study was the usage stage in the cycle. Before the actual usage, implementation and routinization part was introduced to the participants with scenarios. The participants were given with a limited library of gestures to try out according to each scenario.

For the Usage stage, the participants were asked to perform given tasks, then reflect their opinions on the concept. The system was mainly built using wizard of Oz method. At first, Kinect was tried out to build a system that recognizes gestures. However, the use

of one Kinect which was made to be a prototyping level had many constraints. It required the user to stand in the proper position to be detected. Although the detection of gesture was fast enough, gestures had to be performed quite precisely to be detected by the initial system that was built with Kinect. thus, the use of wizard of Oz method was considered more appropriate for this stage of the project, trying out various different options without any restriction of the capability of technologies. See the figure below for the test setup.

Products such as a heater, water kettle, and lamp were connected to arduino which was controlled by a wizard. Each participant was given with instructions by a moderator, since the test cannot follow the realistic time line which would require too much time. The test consisted of three different parts.

1. Use of actions as triggers:Basic concept - Automation based on users’ action

a. A participant was asked to act as if he/she was waking up in the morning. The products were activated by a wizard, as the participant completely stands up from the bed. b. The participant was asked to leave the setup scene and come back, to simulate leaving and coming home.


169


Figure. Wizard-of-Oz test setup

Figure. The use of Arduino for prototyping

2. Use of Activation gesture + deictic(pointing) gesture :Master Control 1 on/off

a. The participant was asked to pick his/her one activation gesture. The participant could either pick from the given library (which included arm raising, arm waving, hand waving, finger snap, and holding index finger for 3 seconds) or come up with anything that he/she would like to use. Thereafter, the participant was asked to turn on/off the devices make deviations from an automated routine.

3.Use of iconic gestures:Master Control 2, natural gesture as input methods

a. At last, the participant was asked to control/adjust devices such as a heater, lamp, or audio (using an audio system was not considered important for the concept, but the related gestures to control an audio system were thought to give insights, thus included in this test).

For each of the test, the evaluation criteria included effort of learning, ease of use, naturalness, intuitiveness and any concerns or suggestion of the concept idea.

170

RESULTS AND DISCUSSION

1.Use of actions as triggers The participants showed positive opinions towards the basic concept, which is to use users’ behavioral actions to trigger daily life tasks. However, there were doubts in system how far the system will be able to detect users’ behavior properly. Deviation was another concerning issue, since morning scenario may be different everyday for some people.

2.Use of Activation gesture + deictic(pointing) gesture In such a system, master control was a necessary feature. However, using activation gesture then deictic gesture was not considered to be very natural. Instead, it was suggested that deictic gesture has to come first to select which product to interact with, and then performing a customized master gesture to actually trigger the task. Once participant mentioned the possibility of using voice, instead of using a pointing gesture to select an object to control.

3.Use of iconic gesturesGestures like shivering, fanning, cupping an ear were considered intuitive and fairly easy to learn for the test participants. It was mentioned by the participants that shivering and fanning were the most natural, because those will be the reaction that they would do unconsciously, and it would be nice if any smart system can recognize their feeling and make changes in temperature. but the others (covering ears, waving, and covering eyes) were considered not natural or intuitive at all. Especially ‘covering ears’ gesture evoked a very awkward experience, since it could be translated into ‘I do not want to listen to you.’, if the gesture was made during

a conversation with someone else. And when it comes to these types of gestures, there can be differences in the form(gesture), expressiveness based on gender and cultural difference.

In the meanwhile, rolling up and down the blinds were considered still natural. Although it is rather more abstract gesture and does not convey any concrete meanings by itself, when mapped into the functionality of the blinds (rolling up and down), it could be used in a natural manner, and it did not raise any weirdness.

PARTICIPANT #1 / 2015.10.15

Feeling natural

Intuitiveness Ease of remembering

Easy for children

Comments

User 1 2 4 3 TEST#1 ACTION BASED TRIGGER Multiple devices How did you feel about the feature that reads your waking up-movement patterns, and activates products that you need especially in the morning? Security How did you feel about the feature that manages your appliances while you and your family members are away from home, by detecting your movement-location pattern? Do you have any concerns? How much effort do you think you are putting into, to use such a system? How natural do you think such a system is? TEST#2 MASTER CONTROL USING GESTURES (Multiple devices) Activation Gesture How natural does the feature feel? How intuitive do you think the activation/pointing gestures are? (is it self-explanatory, mapped well with its functionality?) How easy is it for you to remember the activation/pointing gestures? How easy do you think for children to remember the gestures? if not natural/intuitive/easy, what kind of gesture would you use? What kind of changes would you like to make? Deictic Gesture

- It feels like Harry potter. TEST#3 EMBLEMATIC GESTURE Thermostat How natural does each gesture feel?

- The gesture feels very natural, since both of the gesture is something that you would do when you are actually feeling hot/cold,and right before controlling the thermostat.

How intuitive do you thing each gesture is? (is it self-explanatory, mapped well with its functionality?)

- It feels intuitive too. How easy is it for you to remember all the gestures (with their functionalities)?

- Easy. How easy do you think for children to remember the gestures?

- Easy probably. If not natural/intuitive/easy, what kind of gesture would you use? What kind of changes would you like to

make? Audio/Visual How natural does each gesture feel? How intuitive do you thing each gesture is? (is it self-explanatory, mapped well with its functionality?) How easy is it for you to remember all the gestures (with their functionalities)? How easy do you think for children to remember the gestures? If not natural/intuitive/easy, what kind of gesture would you use? What kind of changes would you like to make? Lighting How natural does each gesture feel? How intuitive do you thing each gesture is? (is it self-explanatory, mapped well with its functionality?) How easy is it for you to remember all the gestures (with their functionalities)? How easy do you think for children to remember the gestures? If not natural/intuitive/easy, what kind of gesture would you use? What kind of changes would you like to make? Shading How natural does each gesture feel? How intuitive do you thing each gesture is? (is it self-explanatory, mapped well with its functionality?) How easy is it for you to remember all the gestures (with their functionalities)? How easy do you think for children to remember the gestures? If not natural/intuitive/easy, what kind of gesture would you use? What kind of changes would you like to make?

PARTICIPANT #2 / 2015.10.15 Feeling

natural Intuitiveness Ease of

remembering Easy for children

Comments

User 1 1 1 1 TEST#1 ACTION BASED TRIGGER Multiple devices How did you feel about the feature that reads your waking up-movement patterns, and activates products that you need especially in the morning? Do you have any concerns?

- Not following the same routine everyday, deviations. For example, to have a breakfast and not to have a breakfast in the morning.


171


Security How did you feel about the feature that manages your appliances while you and your family members are away from home, by detecting your movement-location pattern?

- Like the feature, although I always check everything myself, and I would like to do so. It is like an extra security.

- I don’t want to rush, but take my time in the morning. Do you have any concerns?

- Depends on the product. Alerts and notifications for something not working properly will be very annoying, especially in the morning. For example, my water kettle not containing water inside, or some things out of battery, or low in battery, and everything crashing all together sometime.

- I want no questions, no bothering. If there is no water, it should not be starting then alarming. (morning calm)

- I don’t believe in a system that can detect everything. - What if I have a pet walking around? - Like in the test, I prefer a system that does not have feedback when things are starting. It has to be

there ambient. - Ambient starting preferred.

How much effort do you think you are putting into, to use such a system? - It is like a ghost system, very ambient.

How natural do you think such a system is? - Maybe feeling weird at first, but it will become natural eventually.

TEST#2 MASTER CONTROL USING GESTURES (Multiple devices) Activation Gesture How natural does the feature feel?

- Activation gesture feels very artificial. The order feels wrong. It should be pointing first and activate. Activation gesture is not natural at all. Pointing as well. Maybe eye focusing is better. Not intuitive.

How easy is it for you to remember the activation/pointing gestures?

- It won’t be so easy in the morning. Would I be able to remember my activation gesture in the sleep morning?

How easy do you think for children to remember the gestures? if not natural/intuitive/easy, what kind of gesture would you use? What kind of changes would you like to make? Deictic Gesture TEST#3 EMBLEMATIC GESTURE Thermostat How natural does each gesture feel? For both of the function, it feels very natural. How intuitive do you thing each gesture is? (is it self-explanatory, mapped well with its functionality?)

- It might be better, in some other situation maybe. But it is intuitive. How easy is it for you to remember all the gestures (with their functionalities)?

- Very easy. How easy do you think for children to remember the gestures?

- Easy probably.

If not natural/intuitive/easy , what kind of gesture would you use? What kind of changes would you like to make? Audio/Visual How natural does each gesture feel?

- Gestures make sense, but they don’t feel very natural. It can mean different things at different context.

- The use of the gestures can be limited and unnatural, especially when having a conversation with someone else and covering up your ears. It can feel silly when alone. Speaking can be more intuitive way to control.

How intuitive do you thing each gesture is? (is it self-explanatory, mapped well with its functionality?)

- It can be easily messed up. It does not feel intuitive. How easy is it for you to remember all the gestures (with their functionalities)?

- 2, A bit difficult. How easy do you think for children to remember the gestures?

- For children, it might not work. Because they would cover their ears all the time! If not natural/intuitive/easy, what kind of gesture would you use? What kind of changes would you like to make?

- Turning a knob clockwise, or counter clock wise. more universal gesture, although it could have a lot of context.

Lighting How natural does each gesture feel? How intuitive do you thing each gesture is? (is it self-explanatory, mapped well with its functionality?)

- The least intuitive. How easy is it for you to remember all the gestures (with their functionalities)? How easy do you think for children to remember the gestures? If not natural/intuitive/easy, what kind of gesture would you use? What kind of changes would you like to make?

- The eye glaring gesture can mean ‘zoom-in’. Shading How natural does each gesture feel? How intuitive do you thing each gesture is? (is it self-explanatory, mapped well with its functionality?) How easy is it for you to remember all the gestures (with their functionalities)? How easy do you think for children to remember the gestures? If not natural/intuitive/easy, what kind of gesture would you use? What kind of changes would you like to make? Are there any extra comments, or concerns?

Figure. The user test scene

172

USER TEST - PHASE #2 MASTER THESIS PROJECT: GESTURE CONTROL IN A HOLISTIC HOME ENVIRONMENT

USE OF MASTER CONTROL / ACTIVATION GESTURE + DEICTIC GESTURE

WHEN COMBINED WITH A POINTING GESTURE AFTERWARDS,MASTER CONTROL ALLOWS YOU TO START/STOP EVENTS INSTANTLY.

ON/OFF MASTER CONTROL

MASTERCONTROL

ACTIVATIONGESTURE

POINTINGGESTURE PRODUCT ON / OFF

AND ?

TEST #3 INTUITIVE CONTROLEMBLEMATIC GESTURE

TEST #1 INTELLIGENT SYSTEMACTION (BEHAVIOUR) GESTURE

USER TEST #1 MASTER THESIS PROJECT: GESTURE CONTROL IN A HOLISTIC HOME ENVIRONMENT

#1 ACTION (BEHAVIOUR) GESTURE

You are lying in bed, sleeping. When the alarm goes on, you wake up, and roll out of the bed, In the kitchen, products will be prepared for you.

You are opening the door. You are leaving your house. Everything in the house, that aren’t supposed to be turned on, will be turned off when you leave.

Your house will be secured.

User Scenario #1

User Scenario #2

TEST #2 MASTER CONTROLACTIVATION GESTURE + DEICTIC GESTURE

MASTER THESIS PROJECT: GESTURE CONTROL IN A HOLISTIC HOME ENVIRONMENT

#3 USE OF EMBLEMS / EMBLEMATIC GESTURE

USER TEST - PHASE #2

THERMOSTAT (HEATER)

FUNTION

Increase the temperature Decrease the temperature

GESTURE

Make ‘cold’ gesture :Shivering

Make ‘hot’ gesture :Fanning




SHADING (BLINDS & CURTAINS)

Make ‘up’ gesture :Palm facing down and moving upwards

Make ‘down’ gesture :Palm facing up and moving downwards

FUNTION

Roll up the blinds Roll down the blinds

GESTURE


ASSESSMENTUSER TEST - PHASE #2

How natural does each gesture feel?

How intuitive do you thing the each gesture is? (is it self-explanatory, mapped well with its functionality?)

How easy is it for you to remember all the gestures (with their functionalities)?

How easy do you think for children to remember the gestures?

if not natural/intuitive/easy , what kind of gesture would you use? What kind of changes would you like to make?

Are there any extra comments, or concerns?

VERY NATURALVERY ARTIFITIAL

VERY INTUITIVEVERY FABRICATED (NOT INTUITIVE AT ALL)

VERY EASYVERY HARD

VERY EASYVERY HARD


173


USER TEST #1 MASTER THESIS PROJECT: GESTURE CONTROL IN A HOLISTIC HOME ENVIRONMENT

#1 ACTION (BEHAVIOUR) GESTURE

WHAT PRODUCTS DO YOU HAVEIN YOUR ?KITCHEN

HEATER BLINDS LIGHT STOVE OVEN

COFFEE MAKERMICROWAVE WATER COOKER TOASTER REFRIGERATOR

DISHWASHER BREAD MAKER RICE COOKER SLOW COOKER BLENDER




LIGHTING

FUNTION

Increase brightness Decrease brightness

GESTURE

Make ‘activating sensor’ gesture :Waving in the air

Make ‘eye-glaring’ gesture :Covering eyes

USER TEST - PHASE #2 MASTER THESIS PROJECT: GESTURE CONTROL IN A HOLISTIC HOME ENVIRONMENT

USE OF MASTER CONTROL / ACTIVATION GESTURE + DEICTIC GESTURE

PERSONALIZE YOUR GESTURE FOR THE MASTER CONTROL.

____________ GESTURE

ON/OFF MASTER CONTROL

STAND & STARE ARM WAVE ARM RAISE CLICK HAND WAVE FINGER SNAPADD YOUR OWN

USER 2 GESTURE USER 3 GESTURE



How natural does the feature feel?

How intuitive do you think the activation / the pointing gestures are?

How easy is it for you to remember the activation / the pointing gestures?

How easy do you think for children to remember/use the gestures?

if not natural/intuitive/easy , what kind of gesture would you use? What kind of changes would you like to make?

Are there any extra comments, or concerns?

VERY NATURALVERY ARTIFITIAL


VERY EASYVERY HARD

VERY EASYVERY HARD


VERY EASYVERY HARD



How did you feel about the feature that reads your waking up-movement patterns, and activates products that you need especially in the morning?

Do you have any concerns?

How did you feel about the feature that manages your appliances and secures your house while you and your family members are away from home, by detecting your movement-location pattern?

Do you have any concerns?

How much effort do you think you are putting into, to use such a system?

How natural do you think such a system is?

Not much at allSo much, burdening

Very NaturalVery Artificial




AUDIO / MEDIA DEVICES

FUNTION

Increase volume Decrease volume

GESTURE

Make ‘listening’ gesture :Cupping an ear (ears)

Make ‘not-listening’ gesture :Covering an ear (ears)

174

E.USER TYPOLOGY

PROGRESS PRESENTATION MASTER THESIS PROJECT: GESTURE CONTROL IN A HOLISTIC HOME ENVIRONMENT

ANALYSIS USER TYPOLOGY

INTERVIEW PARTICIPANTS AS POTENTIAL USERS



Philip

Alexandra

Anders Tolstrup

Connie

Tommy

Lexi

Anders

Camilla

Katrine

NOVICE EXPERT

ACCEPTANCE

USER ACCEPTANCE TO SMART LIVING TECHNOLOGYCRITERIA SMART LIVING EXPERIENCE & LIKELYHOOD OF ACCEPTANCE

• No/Little Smart appliances • Conservative • Futuristic vision • High expectation

• Some/Many Smart appliances • Progressive • Realistic vision • Realistic expectation

APPENDIX E. USER TYPOLOGY

175




Philip

Alexandra

Anders TolstrupConnie

Tommy

Lexi

Anders

Camilla

Katrine

VALUE -DRIVEN FUNCTION-DRIVEN

ACCEPTANCE

EXPERIENCE FOCUSED TASK FOCUSED

USER ACCEPTANCE TO SMART LIVING TECHNOLOGYCRITERIA FOCUS OF BENEFITS & LIKELIHOOD OF ACCEPTANCE

• Family Experience • User Needs • Kids • …

• Convenience • Efficiency • Comfortable



Philip

Alexandra

Anders Tolstrup

Connie

Tommy

Lexi

Anders

Camilla

Katrine

GESTURE CONTROL AND SMART LIVING Needless to say, technology progressives are more open to explore new technologies. Furthermore, they are already quite impressed and satisfied with any type of Smart technology that they have at home. They already seem to be happy customers with the changes that (Smart Living technology) have brought to their lives.

CONSERVATIVE PROGRESSIVE

ACCEPTANCE

USER ACCEPTANCE TO SMART LIVING TECHNOLOGYCRITERIA TECHNOLOGICAL ACCEPTANCE & LIKELIHOOD OF ACCEPTANCE

176



Philip Alexandra

Anders TolstrupConnie

Tommy

Lexi

Anders

Camilla

Katrine

* Driving factors considered: appliances, house chore workloads, housing type, living arrangement, newborn baby, etc.

Alexandra (grandma)

NECESSITY PREFERENCE (Optional)

ACCEPTANCE

USER ACCEPTANCE TO SMART LIVING TECHNOLOGYCRITERIA NECESSITY VS. MATTER OF CHOICE(PREFERENCE) & LIKELIHOOD OF ACCEPTANCE

• More residents • More house chores • Multiple floored housing • Newborn baby

• Less residents • Less house chores/appliances • Simple housing



CRITERIA Technological Acceptance X Orientation

CONSERVATIVE PROGRESSIVE

VALUE

FUNCTION

Philip

Alexandra

Anders Tolstrup

Connie

Tommy

Lexi

Anders

Katrine

VALUE SEEKING HESITATORS

CONSIDERATE CONSUMER

TECHNOLOGY UNBELIEVERS

SMART LIVING HIPSTERS

• People who will tryout anyways

• People who will assess new tech, and buy once they find helpful.

• The hesitaters didn’t even try Smart Living yet, but yet skeptical, worried about values spoiled by new technology.

• Not going to buy anything in any case, any soon.

• Simple life style, less necessity • Influenced by Financial affordability

• Make it good, functionally and emotionally.• Prove what Smart Living can add to

their ‘values’.

• Easier Target

Camilla


177



ANALYSIS MAIN TARGET

‘‘ VALUE SEEKING HESITATORS ’’


ANALYSIS INSIGHTS

1. Gesture control can bring functional benefits in certain situations for everyone • Everyone agrees with the fact that gesture control could be helpful at some extend.

2. Wrong idea of Smart Living hinders people to accept it, make them to pick a conservative side… • and some other factors contribute as well, especially money factors, which cannot be changed by design. • ‘‘More things to learn, makes a life more complicated.’’ : I want to live a simple life. • ‘‘Technologies will interfere my family experience.’’

3. Great user experience / usability should follow, to be accepted in the general crowd. • The main thing is scalable control, incorporating both of the tech-savvies and easy users.

178


ANALYSIS STATEMENT

If gesture control can add to what people value on, while providing more comfort and convenience in terms of Smart Living lifestyle, it can attract not only Smart Living hipsters and considerate consumers, but also value-oriented hesitaters, thus increasing the acceptance of Gesture control. Not only the functional benefits to bring small and big convenience in life, but Gesture control should be able to show how it can create more ‘values’ .

Philip

Alexandra

Anders Tolstrup

Connie

Tommy

Lexi

Anders

Camilla

Katrine

DRAWINGS HERE! :)


INSIGHT VALUES

Then what was it that people(hesitaters) valued on?

VALUES FOR SELF

Sense of Empowerment

Being Healthy

Personal Treasure & History

Emotional Benefits

Meaningful Experience - By Saving Time

VALUES WITHIN FAMILY

More Family Experience

Meaningful Experience - By Saving Time

Safety

Caring about Others

VALUES FOR SOCIETY (neighbors/others)

Collaboration with Others

Caring about Others

VALUES FOR THE WORLD

Sustainability - Save Energy


179



ANALYSIS STATEMENT, Rephrased.

The acceptance of Gesture Control can be increased by starting from targeting Value-Seeking Hesitaters, showing what Gesture Control can create/add values, while also providing more functional /emotional comfort and convenience in the context of Smart Living lifestyle.


VISIONARY IMAGE

180

F.IDEA SKETCHES

APPENDIX F. IDEA SKETCHES

181


182

APPENDIX F. IDEA SKETCHES

183


184

G.INTERACTION PROTOTYPING


IDEATION TASKS AND GESTURES

TASK

Writing/ Searching

You could text with one thumb, but two fingers seem to be much more common.

Two thumbs placed on a screen seems the most conventional.

MOST FEASIBLE GESTURE



The phone is brought closer to an ear. The microphone part of the phone is brought closer to the mouth,to make the voice heard better in a noisy situation.

TASK

Talking(calling) Recording a voice/sound

Brining a phone to an ear.


The same gesture to bring the microphone close to the mouth happens when recording a voice.

However, the phone can be placed on a surface when using a speaker phone during a phonecall, or also when recording voice/sound.

However, the phone can be placed on a surface when using a speaker phone during a phonecall, or also when recording voice/sound.

APPENDIX G. INTERACTION PROTOTYPING

185




The phone can be held vertically with one hand, while maintained stable.

The middle button can be pressed with the other hand.

or the button can be pressed with the thumb from the hand holding the phone as well.

The phone can be held horizontally with two hand as well, while maintained stable.

Similarly the middle button can be pressed with a thumb from one of the hands.

Gestures are similar for taking a video.

TASK

Taking a picture Recording a video

A phone held stable, either vertical with one hand, or horizontal with two hands.




Scrolling vertically is the main gestural interaction when browsing a web, SNS application etc.

Scrolling with one hand interaction is shown in this video.

Scrolling can be done with only one hand, but the other hand can support holding the phone.

TASK

Browsing Scrolling up(or down), in the middle of the screen.


video video

186



TASK

Browsing photos Scrolling left(or right).


The screen can be swiped with a thumb,. The screen can be swiped with a thumb while held by the other hand.

-

video video

When the phone is held with one hand, the screen can be swiped with an index finger of the other hand.

video



TASK

Streaming a video Very similar to taking picture/video gestures, the area in front of the screen is cleared. Slight differences from taking a photo/video gesture are that the phone could be held less stable, and tilted. There is more possibility of an earphone connected to the phone.


The phone can be held with two hands stably. The phone can be held stably with one hand. The phone also can be placed on a surface, tiled and supported with one hand.

The phone also can be placed on a surface, tiled and supported with two hands.


187




TASK

Playing music Picking up a phone in a resting state and plug an earphone in.


The phone is picked up and an earphone is plugged in.



TASK

Remote Controlling Approaching toward seems to be the most persuasive. Using a smart phone as a remote control is different from using an IR remote control. however, gestures for using an IR remote control were thought valuable, for its strong implication for the functionality, accumulated through time.


Aiming also does not matter, but the old IR remote control experience can make this natural. However, this could make it hard to see the phone screen, which is necessary for a smart phone to function.

approaching an object to control is natural because of the old IR remote experience, although wifi connection does not require approaching.

188



TASK

Map / Orientation Waving a phone in 8 figure. Map, orientation apps often requires calibrating. If calibrating gesture can be used to start up those apps and the calibration, it could relieves annoyance of calibrating in midst of using such apps.


video

Occasionally, map app requires calibration. Calibrating gesture can be different on other devices (Android phones).

For iOS, calibration gesture is is to wave a phone in 8 figure.



A password is frequently typed with one finger, that is likely to be a thumb.

TASK

Password A thumb placed on a screen after a long resting.


The phone is likely to have been resting, placed on a surface, or in a pocket before wards.


189




For a usual morning alarm or timer, a phone resting on a surface will ring, then picked up, and switched off.

TASK

Alarm Turning a phone could work to switch of an alarm. Lifting up and down, a bit more subtle gesture, could work for a snooze function.


video

When a phone starts to ring because of an alarm/timer, flipping the phone could mean to switch off or to snooze the alarm.

Or, just lifting up could also mean to switch off, or snooze the alarm as well.



Writing / Searching Calling/Recording

Playing Music

PasswordMap/Orientation Alarm/Timer

Streaming a Video Remote Controlling

Taking Photo/Video Browsing a Page

Browsing Photos

SUMMARY

190

H.PRELIMINARY GESTURE EVALUATION STUDY

USER RESEARCH - Gesture Control concept forSmartphones* Required

INTRODUCTION

This research is about a smartphone, that can be controlled by a set of gestures.Thank you for participating in our survey.Your feedback is important. If you have any questions, please let me know by contacting to: [email protected] Please click ''continue'' to proceed.

CONCEPTIn this research, you will be asked to evaluate various gestures in accordance with a specific functionality.

When using a smartphone, some actions that you make have a very strong connection with the function that we are using. For example, when you are answering a call, it is very likely that you are bringing your close to your face, so the speaker and microphone would be positioned near your ears and mouth respectively.

As such, when such a gesture is performed, this new smartphone, controlled by gesture, knows that you are going to receive the call, and initiate the call without clicking any buttons on the screen. Similarly, you can also initiate a call only by picking up your phone and bringing to your face, then say a name in your contact list.

Likewise, some other gestures also can be associated with different functionalities of a smart phone. From the next page, you will be given with 10 functionalities of a smart phone, each of them related with one or two gestures as a trigger. Please evaluate each of the gestures based on 3 criteria; Intuitiveness, Ease of use, and Social awkwardness.

[1] Intuitiveness: Being intuitive means that a gesture, and a functionality have a strong association. The meaning matches well naturally, rather than random or artificial.

[2] Ease of use: Being easy to use means that the gesture designed is simple, and can be performed engaging less body parts (ex.one hand) with less muscle movements. [3] Social awkwardness: Being 'not' socially awkward means that it is very natural, subtle and not awkward to perform a certain gesture in the public, or even alone. Please click ''continue'' to proceed.

MAKE CALLThis gesture is designed to initiate ''Making a call'' function.

1. *Please evaluate the gesture based on the criteria shown below. How do you feel about'brining a phone near your ear' to 'make a call'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


SEND A MESSAGETwo gestures shown below are designed to initiate ''Send a (text) message'' function.

Activation gesture: Bring a phone near your ear (and lips)

http://youtube.com/watch?v=Uzr_5Q9N52c

Activation gesture: Placing two thumbs on a screen

2. *Please evaluate the gesture based on the criteria shown below. How do you feel about'placing two thumbs on a screen' to 'send a message'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


TAKE A PHOTO/VIDEOTwo gestures shown below are designed to initiate ''Taking a photo/video'' function.

Activation gesture: Holding a phone stable with two hands

3. *Please evaluate the gesture based on the criteria shown below. How do you feel about'holding a phone stable with TWO hands' to 'take a picture/video'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


4. *Please evaluate the gesture based on the criteria shown below. How do you feel about'holding a phone stable with ONE hand' to 'take a picture/video'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


RECORD A VOICE/SOUNDThe shown below is designed to initiate ''Record a voice'' function.

Activation gesture: Holding a phone stable with one hand

Activation gesture: Bring 'the microphone' componentnear your mouth

5. *Please evaluate the gesture based on the criteria shown below. How do you feel about'brining the microphone component close to the mouth' to 'record a voice/sound'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


PLAY MUSICThe gesture shown below is designed to initiate ''play music'' function.

Activation gesture: Connecting an earphone and put thephone aside

http://youtube.com/watch?v=X9Yc7FVeH1Y

6. *Please evaluate the gesture based on the criteria shown below. How do you feel about'connecting an earphone' to 'play music'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


STREAM A VIDEOTwo gestures shown below are designed to initiate ''Stream a video'' function. Please note that the phone is sitting on a surface.

7. *Please evaluate the gesture based on the criteria shown below. How do you feel about'placing a phone on a surface, supporting with ONE hand' to 'stream a video'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


Activation gesture: Placing a phone on a 'surface',supporting with one hand

Activation gesture: Placing a phone on a 'surface',supporting with two hands

APPENDIX H. PRELIMINARY GESTURE EVALUATION SURVEY

191


8. *Please evaluate the gesture based on the criteria shown below. How do you feel about'placing a phone on a surface, supporting with TWO hands' to 'stream a video'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


ENTER A PASSCODEThe gesture shown below is designed to initiate ''Type a passcode'' function.

Activation gesture: Picking up a phone from a 'sleep(rest)' state.

http://youtube.com/watch?v=U83WeBP33GI

9. *Please evaluate the gesture based on the criteria shown below. How do you feel about'picking up a sleeping phone' to 'enter a passcode'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


REMOTE CONTROLThe gesture shown below is designed to initiate ''Remote control'' function.

10. *Please evaluate the gesture based on the criteria shown below. How do you feel about'approaching forward' to 'remote control'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


MAP/ORIENTATIONThe gesture shown below is designed to initiate ''Map'' function.

Activation gesture: Approaching forward

http://youtube.com/watch?v=OCYbhd0k8Bg

Activation gesture: Waving in the air in 8 figure

11. *Please evaluate the gesture based on the criteria shown below. How do you feel about'waving in 8 figure' to 'open a map'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


TURN OFF AN ALARM/TIMERThe gesture shown below is designed to initiate ''Turn off an alarm/timer'' function.

http://youtube.com/watch?v=FyAHdbx1PpM

Activation gesture: Flipping a phone

http://youtube.com/watch?v=31LTWslEvo8

Powered by

12. *Please evaluate the gesture based on the criteria shown below. How do you feel about'flipping a phone' to 'turn off an alarm/timer'?Mark only one oval per row.

Very poor(1)

Poor(2)

Neutral(3)

Good(4)

Very good(5)


The endThank you so much! This is the end of the research. Before you go, please let us know who you are.

13. Full name *

14. Email address

(45 responses)

RECORD A VOICE/SOUND

(45 responses)

PLAY MUSIC

(45 responses)

STREAM A VIDEO

(45 responses)

2

4

6Very poor (1)Very poor (1) Poor (2)Poor (2) Neutral (3)Neutral (3) Good (4)Good (4) Very good (5)Very good (5)

Intuitiveness (Very artificial - Veryintuitive)

Ease of Use (Very hard - Very easy) Social awkwardness (Very awkward -Very natural)

0

2

4




0

2

4




0.0

2.5

5.0

7.5 Very poor (1)Very poor (1) Poor (2)Poor (2) Neutral (3)Neutral (3) Good (4)Good (4) Very good (5)Very good (5)

15 responses

Accepting responses

INTRODUCTION

CONCEPT

MAKE CALL

(45 responses)

SEND A MESSAGE

(45 responses)

TAKE A PHOTO/VIDEO

(45 responses)



0.0

2.5

5.0

7.5Very poor (1)Very poor (1) Poor (2)Poor (2) Neutral (3)Neutral (3) Good (4)Good (4) Very good (5)Very good (5)



0.0

2.5

5.0


FINAL RESULT USER RESEARCH - Gestur

QUESTIONS RESPONSES

(45 responses)

ENTER A PASSCODE

(45 responses)

REMOTE CONTROL

(45 responses)

MAP/ORIENTATION



0.0

2.5

5.0




0.0

2.5

5.0




0

4

8




0.0

2.5

5.0

7.5


Email address (15 responses)

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

MAP/ORIENTATION

(45 responses)

TURN OFF AN ALARM/TIMER

(45 responses)

The end

Full name (15 responses)

Johannes Rasmussen

Kat Nunn

Martin Moi

Anders Tolstrup Rasmussen

Jennifer Wong

Sara Eszter Kis

Jihoon Lee

Koonlada Sasamit

Tal

Sara Engvild

Per Nordström

Chun Wei Chang

Youngin Park

Mereike

Alice Smejkalova



0.0

2.5

5.0




0.0

2.5

5.0

7.5


192

[Sara Ester Kis] Female, 26, Hungary I like the general idea of the concept, except for the passcode function.

FUNCTIONALITY FEEDBACK

MAKE A CALL

SEND A MESSAGE

TAKE A PHOTO/VIDEO

One hand


PLAY MUSIC

STREAM A VIDEO

Two hands

ENTER A PASSCODE Fingerprint is the easiest, so I don’t even have to type a passcode afterwards.

REMOTE CONTROL

MAP/ORIENTATION

TURN OFF AN ALARM

[Mareike Bonitz] Female, 23, Germany


MAKE A CALL

SEND A MESSAGE

TAKE A PHOTO/VIDEO

One hand I do this gesture a lot, when I am doing this. if i get a photoshot function everytime when i do this gesture, it will be annoying. could get the wrong functionality. if you get the photo function when not intended, its awkward.

RECORD A VOICE/SOUND I never record anything, but the gesture , i would do it when i put my earphone.

PLAY MUSIC

STREAM A VIDEO This gesture using one hand,, i always do this gesture whenever I read news. I read news horizontally, to have bigger texts...

Two hands This case, using two hands is a bit more special gesture, so not confusing with any other functionality.

ENTER A PASSCODE

REMOTE CONTROL

MAP/ORIENTATION

TURN OFF AN ALARM There is not so much difference with clicking a button, or swiping. but i like the part that i don't have to see. but just shutting it off without looking at it.

[Chun Wei Chang] Female, 27, Taiwan


MAKE A CALL

SEND A MESSAGE I type with one hand most of the time. But it is still okay.

TAKE A PHOTO/VIDEO

One hand

RECORD A VOICE/SOUND Most of the times, recording others’ voice. sometimes recording my own voice, put the phone down on the table afterwards. like serious case, discussion, meeting etc. Does everyone know where the microphone is?

PLAY MUSIC

STREAM A VIDEO I will do such a gesture in many other contexts maybe. but maybe not on a surface! But I would now like to change my scores to (5)-(3)-(5)

Two hands Two hands part, it is not very user friendly. Can I change my scores to (5)-(2)-(4)?

ENTER A PASSCODE (4)-(5)-(5)

REMOTE CONTROL It is funny, and feel awkward, because i personally never had to do this before.

MAP/ORIENTATION

TURN OFF AN ALARM maybe i can push it away as well, to snooze the alarm.

[Youngin Park] Female, 25, Korea I have lower scores for the Ease of use when I have to use two hands when I can also do the same tasks with only one hand. Also for the occasions that I have to put some physical efforts, such as going against gravity.


MAKE A CALL

SEND A MESSAGE

TAKE A PHOTO/VIDEO I gave a lower score for Ease of use, because occasionally, I don’t use both hands to take a picture. There is always a benefit to reduce many in-between steps to reach a goal.

One hand This gesture is quite general, It could conflict with other tasks with similar gestures.

RECORD A VOICE/SOUND I thought lifting up, and bringing close were an extra work. And the task of recording itself makes it awkward. It always draws attention, since it is not a task that you do often.

PLAY MUSIC It is hard to find earphones! although I always manage to do so anyways. but just comparing to other gestures and tasks, it has more steps I feel like.

STREAM A VIDEO This is better than the other option of using two hands.

Two hands I don’t like the idea of using two hands.

ENTER A PASSCODE So you are turning on the screen by lifting a phone? It is not hard at all. I will give 4 points.

REMOTE CONTROL It said that you have to push forward, approach forward, but it is a bit tricky. Pushing (approaching) means that you have to bring it back as well. maybe it could be shaking? Approaching somehow feels like I need to concentrate. To make it natural… I don’t know. but If I do this, I think people will be curious of what I am doing. I can be eye-catching.

MAP/ORIENTATION This is familiar, and makes sense to me, since I had to do a lot, but I never never felt it was easy. It is hard and embarrassing. I mean everyone can now see I am lost and looking at my map to find a way, and my map is not even working well.

TURN OFF AN ALARM When it comes to Ease of use, or usability, I always sleep with my phone already flipped over. when this gesture does not have any directionality, It would be fine. but when it always has to start from the screen-top position, it will be tricky. I will think, ‘do I always need to do this way?’


193


[Joanne Lee] Female, 26, Korea


MAKE A CALL

SEND A MESSAGE This gesture can prompt a lot of different apps, I think.

TAKE A PHOTO/VIDEO I do take photo or video with two hands, supporting it with two hands. but I could do the same gesture to play some games. if camera function pops up when I want to play a game, it would be frustrating.

One hand I hardly take a photo with one hand. iPhone 6s is too large to do so. but then it was also the same when i was using iphone 5s.


PLAY MUSIC I could plug in an earphone for different situations, to call, for example.

STREAM A VIDEO I don’t really use my phone with one hand. if I do, i would usually lean the phone on something else. when i watch a video, i am holding my phone with two hands most of the time. I would do it with one hand from time to time, but i wouldn't put it on a surface.

Two hands

ENTER A PASSCODE I use TouchID a lot. I think it is still a good idea to turn on the screen by lifting it up.

REMOTE CONTROL Approaching forward feels just awkward. I don’t even do this with an IR remote controller at home. I only do it when it does not work well.

MAP/ORIENTATION I just don’t like this action - although i have to do this from time to time. Its is not even essential to get the map, but only for calibrating for the compass.

TURN OFF AN ALARM If I get an alarming sound, and I don’t have to see the screen, I just shortly push the power button. If I get to see the screen, or in the occasion that I actually see the screen, it will be nice if the screen turns off when I see and cover, or flip it over. If i don't have to see the screen, and not even know how my phone is place (either screen up, or backside up) I will just press the lock button. If i consider this gesture as an ‘additional function’, If I get to flip it over, it should be the case that I already read my screen and turn off. If it can be something that I don't even have to see, the button is more convenient. you already have a button anyways.

[Kat Nunn] Female, 22, UK


MAKE A CALL Cannot think of anything else easier

SEND A MESSAGE I normally type with one thumb, right handed, and type with my right hand. and sometimes go with two again. my phone is quite big. but for one finger, i would like to reserve it for other function, that i am more likely to do with one finger. and it's like texting, i dont have to feel awkward at all.

TAKE A PHOTO/VIDEO I would normally take a picture with one hand - but i get blurry pictures. I should probably take pictures with two, but it isn't that easiest compared to one hand.

One hand One hand is easier, but I have this weird gesture + click the thumb with one hand. you don't get the picture as stable as the first one though.

RECORD A VOICE/SOUND I hold it like normal way, don’t really flip it. it is not what i am doing now, i am used to do it in the other way, but it is something i can start doing. and that is where my microphone is. The gesture is fine, and it is a bit awkward just to talk in public to prompt a recording gesture. the functionality itself is weird.

PLAY MUSIC It is really easy to use. but i can use the normal interface, as well, right? i don't think i put my earphones in for watching a youtube video, but only with music. I think you just have to play what pay you wanted - I use googleplay!

STREAM A VIDEO I could learn to do it, if i have to, but i am usually just holding it. Because you have to type on your screen, to choose a video on. initiating it by lifting up… is weird. but it is not socially awkward though. I would do this in public, but just slightly not intuitive.

Two hands I would never do this. I wouldn't’ be embarrassed though, not at all, but why would i do this gesture?

ENTER A PASSCODE I don’t have a passcode. isn't the iphone have this swipe, so it wouldn't be accidently turned on? worried about battery life, accidently turned on. the actual gesture is intuitive though, obviously i pick up my phone to start. the gesture is easy to do. for me, i need a swipe because i don't have a passcode.

REMOTE CONTROL This isn’t something that i have on my phone. but it is intuitive, if i ever remote control my app. it is so easy to click and turn on my app with my thumb, so i would usually do it. i am indifferent. it is not awful.

MAP/ORIENTATION I have never asked to do this. I would just drop my phone. I think it looks very unnatural.

TURN OFF AN ALARM People do that to get rid of the phone call, i think it is very cool feature. also for the alarm. making it ‘quite’ make sense. maybe clicking the button is easier than flipping. compared to pressing a button, it is a bit hard, but it is still easy. and completely normal.

[Alice Smejklova] Female, 24, Czech Republic


MAKE A CALL

SEND A MESSAGE it’s much easier to type with one finger. and I usually work with my phone with one hand.. faster for me to press an app button. it would be faster if i could do that from the lock screen!

TAKE A PHOTO/VIDEO For taking a quick photo, usually with one hand. and I use the camera icon in the lock screen, i use that a lot. That is the most convenient, and quickest way for me to take a picture.

One hand


PLAY MUSIC

STREAM A VIDEO

Two hands

ENTER A PASSCODE

REMOTE CONTROL

MAP/ORIENTATION I feel awkward. if it would be implemented, people would know what I am going. but funny.

TURN OFF AN ALARM

[Anders Tolstrup Rasmussen] Male, 26, Denmark

- Consider adding ‘MUTE’ function. For iphones, one side button is the only control for the function, and mine got broken and now I cannot mute normally at all.


MAKE A CALL

SEND A MESSAGE I don't really see the connection between putting two thumbs and sending a message. I usually do it with one thumb. maybe more with emails, emails are longer?

TAKE A PHOTO/VIDEO I rarely take photos in this way. but this would be still more intuitive than this one. but could be watch a video… etc.

One hand this gesture could be anything start position for everything.

RECORD A VOICE/SOUND awkward. i think its is a gesture. when i do record, i do use the function a lot, but during the interviews.

PLAY MUSIC putting an earphone can be something else as well. a lot of different function. open my spotify, radio app, it would be annoying.

STREAM A VIDEO different gestures for making a video, it is intuitive that the screen is landscape view, and rather than that…

Two hands still doesn't make a bit different, using two hands will be slightly different, but not a lot.

ENTER A PASSCODE

REMOTE CONTROL Pointing at my sonos, and if it can detect, it will be nice. TV or alarm, or whatever, but pointing is both intuitive and cool.

MAP/ORIENTATION gesture should be subtle.. this case is an overkill.

TURN OFF AN ALARM I like that. flip it over and just go back to sleep won't be nice, though.

194

[Martin Moi Lundstad] Male, 28, Norway Most of the thing is not an add on, so it's okay.


MAKE A CALL

SEND A MESSAGE

TAKE A PHOTO/VIDEO

One hand


PLAY MUSIC Audio feedback might be nice. i would always check the time going up. I don't really trust the music to be turn on or off.

STREAM A VIDEO The video thing, don't think i watch that much video. I never flip it. I just hold it maybe. but don't do it, because I just never watch a video with my phone. If I check I have a video file to watch, i will just open this link in my browser.

Two hands

ENTER A PASSCODE when you walk around, how does the passcode work? I never have my phone on the desk. sit with the phone.

REMOTE CONTROL I really like that. i was a bit unsure, because i don't have a controller app.

MAP/ORIENTATION Map gesture is a little awkward. I always thought it was weird.also for calibrating the compass. It is really weird thing to do. it doesn't really work that well…

TURN OFF AN ALARM you will gonna mess up with the alarm in any case, when waking up. it is snooze? or turn off?

[Johanness Rasmussen] Male, 27, Denmark


MAKE A CALL

SEND A MESSAGE

TAKE A PHOTO/VIDEO I could watch a video in the same position. play a game, read, etc. I could be doing anything with the phone.

One hand

RECORD A VOICE/SOUND If it works, it would be good. Although, how does the technology work?

PLAY MUSIC It could be a lot of different things. I might watch a video, or something else.

STREAM A VIDEO Similarly, I could be doing anything.

Two hands I could be doing anything.

ENTER A PASSCODE

REMOTE CONTROL

MAP/ORIENTATION

TURN OFF AN ALARM

[Koonlada Sasamit] Female, 28, Thailand


MAKE A CALL Pick up is fine, but calling might be weird. especially when it's getting the name wrong. it's more about the voice part. I don’t really trust the system somehow. because i don't really know who's on the phone, on screen, and cannot see the screen, i might feel a bit insecure.

SEND A MESSAGE if not pressing, but locating the thumbs on a screen to be ready for typing, it is good. not awkward.

TAKE A PHOTO/VIDEO holding with two hands don't really work.. people definitely knows that i am taking a picture, it is not discreet. hold and pressing the button at the same time. I think as a shortcut to use the camera, it will be useful! but i prefer this is not the only way - most of time i don't use two hands. i am thinking of buying a smaller phone.

One hand I think it's fine, and it feels better than using two hands.

RECORD A VOICE/SOUND if i am trying to record my own voice, but no. but i never record my voice so i didn't think about that possibility. if i have to record somebody else's voice, i will just hold it upside down. putting the microphone UP at least for me, is natural.

PLAY MUSIC I think it is very common. I wouldn't notice anyone care, weird, inappropriate or awkward. it is very natural.

STREAM A VIDEO I need to use some imagination, since I never watch a video on my phone.

Two hands I wouldn't do this. It is not natural for me, to activate. I wouldn't think of any activating gesture. I would just go with touch screen. for both one hand and two.. I don't use two hands that much with my phone.

ENTER A PASSCODE It is straightforward, if my phone can react based on my movement.

REMOTE CONTROL

MAP/ORIENTATION This is the weirdest gesture ever, I have never done this.

TURN OFF AN ALARM

[Sara Engvild] Female, 33, Denmark It feels intuitive, if it is already part of what I am doing.


MAKE A CALL It is easy, only except for sometimes when I am on a bike. When cycling, nothing is easy at all to pick up etc.

SEND A MESSAGE It is intuitive, and using two thumbs to go to an app is okay. easy to use. it is also socially not awkward at all.

TAKE A PHOTO/VIDEO The gesture is easy, but it would not be easy to hold still and stable with one hand. it won’t be awkward, but it could depend on a situation as well.

One hand especially when take a horizontal photo, i will do it with two hands. the more discrete the gesture is, i think i will feel less awkward.

RECORD A VOICE/SOUND it depends on a situation. when alone at home, it would not be awkward. but when with other friends, or people it will be awkward. Recording activity itself is weird. it is easy and intuitive, but it could be easier though.

PLAY MUSIC Quite intuitive, the gesture makes sense with the functionality. But if it is easy to use, it depends on my situation such as cycling or not. when cycling, all the usability is poor. it is not awkward at all.

STREAM A VIDEO I don’t watch a video like the given gesture. I usually hold my phone and watch something. so this is not intuitive. but it is also not hard when it comes to ease of use. it is also not awkward. I think in general (except for taking a picture) using one hand is a better option.

Two hands I would never do this with two hands.

ENTER A PASSCODE Really good. I did not know it was only for turning on the screen, i thought it was typing the passcode. normally, I would use the home button to turn on the screen.

REMOTE CONTROL Really good. makes sense to me, and easy to use. Pointing really makes sense. more intuitive that just navigating things in the phone screen. it works like how it used to be, like an old remote!

MAP/ORIENTATION I have no experience with this gesture. thus very poor for intuitiveness. although it is easy, I mean not that difficult to learn this. but this is very awkward - i cannot imagine doing this in public.

TURN OFF AN ALARM This is not easy with iphone 5, since it has a sharp edge, so it doesn't really feel nice on the edges. It could be easier with iphone 6 series. It is not very intuitive. And I can think of another use. I could use it for other functions, for example if i am in a meeting with people, i might do the same gesture when i am getting irrelevant messages and i would like to make it discreet from other people.


195


[Tal Amram] I like the general idea of the concept, except for the passcode function.


MAKE A CALL

SEND A MESSAGE One or two hands depending on the situation, but I don't open an app with two thumbs. Anyways, this is my first reaction to seeing the videos and images.

TAKE A PHOTO/VIDEO

One hand


PLAY MUSIC

STREAM A VIDEO

Two hands

ENTER A PASSCODE

REMOTE CONTROL I don’t think I gave a low score for this function, and I think I doubted if I could see the phone screen when I would hold it as a remote contro l.

MAP/ORIENTATION

TURN OFF AN ALARM

196

PILOT USER TEST PROCEDURE AND REFLECTION

OBJECTIVEBy so far, functions are already mapped with gestures. Thus, rather than being explorative, the study focuses on evaluating the concept that came up and polished so far. Along with the criteria from the previous study to assess gestures as a control method, criteria related to smart living were added again, considering the new remote control feature used in home environment setup.

RESEARCH QUESTIONA technical prototype to assess gesture recognition cannot be built, and also gesture recognition of the system, or the detailed feasibility of the system is not the main concern. I would like to focus on the correctness of the mapping, being in-line with Smart living, and superiority/complementariness of gesture control in certain situations. Furthermore, I would like to verify the appropriateness of the feedback designed for the system.

Gesture controlGesture• How intuitive is the control (mentally easy to remember)?• Did the participants remember the gesture, and use it correctly?• How easy is the control (physically easy to perform, considering motor

skills, complexity of the gestures etc.?)• How awkward is the control, creating social awkwardness, within a given

environment (home)?• Would it make any difference if performed in public?

Feedback• How adequate is the vibration for each step of the control? (enough clue

of what is happening?)• How helpful is the vibration, to understand what is happening?• How helpful is the visual feedback, to understand what is happening?• How easy is it to remember the meaning of the (both) feedback?

Smartness (Smart Living)Effort-saving• Does it feel like a shortcut (distance wise)?• How do you feel about controlling your appliances with gestures?• How do you think about the duplication of the settings by drag-and-drop?

Expressiveness• How do you think about the expressiveness of the gesture control?

METHODParticipantTwo Master students studying Design for Interaction were invited as a test participant and a wizard for the test. Criteria to consider in the real test: Age, Handedness, Education background, prior experience

with Smartphone and related gadgets, prior experience with gesture control, and nationality.

Preparing the sceneThere are three main components to be tested. The gestures, and two feedbacks which includes vibration and blinking on the target object. The core functionality which is to recognize gestures are done by Wizard-of-Oz, considering the complexity of the subtle gesture recognition, the difficulties of developing an application that works with iPhone, and the limited time.

Since gesture recognition is only simulated, the corresponding feedback should be simulated accordingly as well, which provides multiple problems such as having too many wizards, one for the main function, another one for the vibration, and the other for the visual feedback. Thus, instead of conducting a study in a real environmental setting, a living room scene was projected to simulate the environment. (line drawing was used for the ease of Photoshop work only for the pilot test.) In this way, the functionality and visual feedback could be handled by one wizard, in the projection. The vibration feedback was simulated by another wizard separately for this pilot test, which is planned to be integrated in the future user test.

ProcedureThe participant was given with instructions about how the gesture control works. Then he was asked to interact with a curtain, blinds, and light respectively, one at a time. As he performs a gesture, two wizards controlled the scene so it would play a desired effect. After this, the participant was asked for the opinions about the session.

FINDINGSSyncing issueIt was hard to sync two wizard’s input. It would be ideal if functional control-haptic feedback (vibration)- visual feedback (LEDs) are integrated together.

Environment settingIt would be better, if the size of the projection is bigger so the aiming points are more clear to the wizard. Also if it can be projected from the floor level, or projected to the wall directly, it might be able to simulate more natural interaction rather then looking at all the appliances locating at an eye-level. Fortunately, the participant and the wizard mentioned that having a projection instead of real scene would not affect their evaluations much. However, both mentioned that a wizard will have to perform a good job during the real test in the future. It was recommended to use a real picture instead of a line drawing, if the objects can be photoshopped nicely.

Use of Smart phoneFor a better grip of the remote control, consider using a small box and put the vibration sensor inside of it to substitute a real smart phone with the vibration module attached outside.

I.PILOT STUDY

APPENDIX I. PILOT STUDY

197


REVISE OF THE PLANBuilt something with Arduino and connect it with PC, so function and feedback can be integrated into one so only one wizard is needed. Gestures can be presented in a form of video. Prepare

survey/assessment criteria. Prepare an interactive screen – build with either Flash, Processing, or Axure. For now, Processing and Flash seem feasible to connect easily with Arduino input, especially Processing. Somehow, Axure gave me hard time simulating curtain/blinds effect.

198

INTRODUCTION

Simply introduce the project, and the objective of this study.

“Thank you for your participation. I am working on a thesis to come up with Gesture controls for smarter living. I came up with gesture sets using a smart phone, to control the smart phone itself and home environments. You will be introduced with the gestures and asked to perform them, and evaluate them afterwards. This study consists of two different parts, and it will take half an hour to an hour.

• I would like to start from asking few details about you.”• How old are you?• What is your nationality?• Do you have a smart phone?• And how actively are you using it, are you an active-mediocre-

novice user?• Do you own any Smart products at home?• Are you right-handed or left-handed?

PART #1 SMART PHONE GESTURE CONTROL

“The first part of this study is about the Smart phone gesture control.”

Procedure1.Show Smart phone gesture control with a prepared video (duration: 2:30)

2.Ask a participant to make a correct gesture to execute/launch a requested function.

3.While closely observing the gestures made by the participant, the wizard controls and triggers the haptic feedback on the vibration module accordingly from the laptop.

4.A short interview session follows afterwards.

5.Mark the result accordingly using the assessment table.

[Interview questions]Pass/fail• Did the participant remember the gesture, and perform it

correctly?

Gesture• How intuitive was the control (mentally easy to remember)?• How easy was the control (physically easy to perform,

considering motor skills, complexity of the gestures etc.?)• How awkward is the control, creating social awkwardness,

at home?• Would you feel and react different comparing when you are

alone and when there are people around you?• Would it make any difference if performed in public?• Would you feel and react different comparing when you are

alone and when there are people around you?

Feedback• How adequate is the vibration for each step of the control?

(enough clue of what is happening?)• How helpful is the haptic feedback (vibration), to understand

what is happening?• How easy is it to remember the meaning of the feedback?

Smartness (Smart Living)• Does it feel like a shortcut (step- wise)?• How do you generally feel about controlling your appliances

with gestures?

J.EVALUATION STUDYGUIDELINE

Table 1. Table for the assessment

APPENDIX J. EVALUATION STUDY GUIDELINE

199


PART #2 REMOTE CONTROL GESTURES FOR HOME ENVIRONMENTS

The second part of the test is to evaluate remote control gestures for controlling home appliances, using a smart phone. Firstly, I will show you another video that explains how it works. Afterwards, I will ask you to control some of the objects in the scene. When you do so, please ‘think aloud’.

Procedure1.Show Smart phone gesture control with a prepared video (duration: 2:30)

2.Ask a participant to make a correct gesture to execute/launch a requested task.

3.While closely observing the gestures and comments made by the participant, the wizard controls the scene and triggers the haptic feedback on the vibration module accordingly, from the laptop.

[Tasks]• Can you open the curtain slowly, while concentrating on the

given feedback?• Can you close the curtain slowly, while concentrating on the

given feedback?• Can you open the roller blinds slowly, while concentrating on

the given feedback?• Can you close the roller blinds slowly, while concentrating on

the given feedback?• Can you turn on the speakers slowly, while concentrating on the

given feedback?• Can you turn up the volume slowly?• Can you turn down the volume slowly?• Can you slowly turn on the first light on the left?• Can you slowly turn off the first light on the left?• Can you slowly turn on the first light, then copy the light to the

second light?• Can you slowly increase the temperature of the thermostat?• Can you slowly decrease the temperature of the thermostat?• Can you open the curtain fast?• Can you close the curtain fast?• Can you adjust it, as you like?• Can you open the roller fast?• Can you close the roller fast?• Can you adjust it, as you like?• Can you adjust the volume of the speaker as you like?• Can you turn on/off the light,• And adjust is as you like?• Can you control the temperature of the thermostat?

4.A short interview session follows.

[Interview questions]Pass/fail• Did the participant remember the gesture, and perform it

correctly?Gesture• How intuitive was the control (mentally easy to remember)?• How easy was the control (physically easy to perform, consider-

ing motor skills, complexity of the gestures etc.?)• How awkward is the control, creating social awkwardness, at

home?• Would you feel and react different comparing when you are

alone and when there are people around you?

Feedback• How adequate is the vibration for each step of the control?

(enough clue of what is happening?)• How helpful is the haptic feedback (vibration), to understand

what is happening?• How helpful is the visual feedback (LED), to understand what is

happening?• How easy is it to remember the meaning of the (both) feedback?• Smartness (Smart Living)• Does it feel like a shortcut (step- wise)?• How do you generally feel about controlling your appliances

with gestures?• How do you think about the duplication of the settings by drag-

and-drop?• How do you think about the expressiveness of the gesture

control?

5.Mark the result accordingly using the assessment table.

Table 2. Table for the assessment

Master thesis: Smartphone Controlled Home Environments

Design

Transcript of Master thesis: Smartphone Controlled Home Environments