Towards standards in user interface design of networked ...1494/fulltext.pdf · Networked...
Transcript of Towards standards in user interface design of networked ...1494/fulltext.pdf · Networked...
Towards Standards in User Interface Design of Networked Devices
A Thesis Presented by
Sanel Selimovic
to
The Department of Mechanical and Industrial Engineering
in partial fulfillment of the requirements for the degree of
Master of Science
in
Industrial Engineering
in the field of
Human Machine Interaction
Northeastern University Boston, Massachusetts
December 2010
II
Abstract Networked appliances have long held promise of making people’s lives easier by
allowing users complete access over their functionality through an intermediate device such as a
tablet computer. Corporations often see no benefits to standardizing user interface across their
multiple appliances, or they feel that their proprietary software offers them a competitive edge in
the marketplace.
This thesis explores the underlying set of principles for home appliances. It attempts to
identify user expectations in terms of what they want to see in their networked appliances in
terms of features and functionality, but also provide guidelines for future developers. To
accomplish this, two prototypes were developed and tested using a System Usability Scale (SUS)
survey and participant success rates were measured in terms of the time it took to complete each
task, and the success rate across participants and across tasks. The two experiments were
conducted using a remote usability tool. Study results indicate statistically significant differences
between the two prototypes.
III
Acknowledgements I would like to thank everyone who has helped me out on this journey. First and foremost I
would like to thank my advisor Yingzi Lin for all of her trust, help and support she has given me
on my thesis work and in my academic work. I would also like to thank my friend and statistical
guru Ahamed Altaboli for all of his help on this and many other projects. Thank you to everyone
else whom I haven’t specifically called out.
My time in Boston, and my time at Northeastern in our lab, has been the most amazing two years
of my life. Once again, thank you Professor Lin for giving me this opportunity.
Sincerely,
Sanel Selimovic
IV
Table of Contents
Chapter 1: Introduction .......................................................................................................... 1 Overview ...................................................................................................................................................... 1 Background .................................................................................................................................................. 2
Motivation ............................................................................................................................................... 2 Simplified UI Methods ............................................................................................................................. 4
Research Problems ...................................................................................................................................... 5
Chapter 2: Conceptual Framework .......................................................................................... 6 Approaches to Simplified UI ........................................................................................................................ 6
Simplifying Consumer Electronics Interfaces .......................................................................................... 7 Mixed Initiative (Collaborative) Interfaces .............................................................................................. 9 Goal‐Oriented Interfaces ....................................................................................................................... 10 Other Interaction Modes and Guiding Principles .................................................................................. 11
System Usability Scale ............................................................................................................................... 12 Error Rates ................................................................................................................................................. 13 Task Completion Times .............................................................................................................................. 14
Chapter 3: Experiment Design ............................................................................................... 14 Hypothesis ................................................................................................................................................. 14
Definition ............................................................................................................................................... 15 Performance Data ................................................................................................................................. 15 Task Completion and Error Rates .......................................................................................................... 16
Experiment Setup ...................................................................................................................................... 16 Introduction .......................................................................................................................................... 16 Initial Survey .......................................................................................................................................... 17 Prototype 1 Setup ................................................................................................................................. 19 Prototype 2 Setup ................................................................................................................................. 25 Scoring the System Usability Scale (SUS) .............................................................................................. 30
Experiment Procedure ............................................................................................................................... 32 Task List ................................................................................................................................................. 35
Chapter 4: Experiment Results .............................................................................................. 36 Initial Survey Results .................................................................................................................................. 36 System Usability Scale (SUS) Results ......................................................................................................... 43
SUS Results Analysis .............................................................................................................................. 43 Performance Data Results ......................................................................................................................... 46
General Results ..................................................................................................................................... 46 Statistical Analysis ................................................................................................................................. 53 Overall Summary ................................................................................................................................... 56
Chapter 5: Conclusion ........................................................................................................... 56
Appendix .............................................................................................................................. 58 Appendix A: SUS Survey ............................................................................................................................. 58 Appendix B: Instructions to Participants ................................................................................................... 59
References ............................................................................................................................ 60
V
Table Index Table 1: Summary Of Participant Interest In Remote Control Of Given Appliances .................................. 18
Table 2: List Of Tasks And Order They Were Presented To The User ........................................................ 35
Table 3: Breakdown Of Participant Responses To Short Survey ................................................................ 37
Table 4: Most Frequently Requested TV Features And Functionality ....................................................... 39
Table 5: Summary Of Responses For The A/C ........................................................................................... 40
Table 6: Summary Of Responses For Heater ............................................................................................. 40
Table 7: Summary Of Responses For Lights ............................................................................................... 41
Table 8: Summary Of Responses For DVD Player ...................................................................................... 41
Table 9: Summary Of Responses For Surround Sound .............................................................................. 42
Table 10: Summary Of Responses For Home Security ............................................................................... 42
Table 11: Summary Of Responses For Door Locks ..................................................................................... 42
Table 12: Prototype One ‐ Summary Of Responses To The SUS Survey .................................................... 44
Table 13: Prototype One ‐ Summary Of Statistical Data For SUS Survey ................................................... 44
Table 14: Prototype Two ‐ Summary Of Responses To The SUS Survey .................................................... 45
Table 15: Prototype Two ‐ Summary Of Statistical Data For SUS Survey .................................................. 45
Table 16: Average Task Completion Time ................................................................................................. 46
Table 17: Average Number Of Errors Across Participants And Across Tasks For Experiment 1 ................ 48
Table 18: Average Number Of Errors Across Participants And Across Tasks For Experiment 2 ................ 50
Table 19: Experiment 1 – Mean Task Ease Of Use Ratings ........................................................................ 51
Table 20: Experiment 2 – Mean Task Ease Of Use Ratings ........................................................................ 52
Table 21: Average Number Of Seconds To Complete A Task .................................................................... 53
Table 22: Unpaired T‐Test Of Average Completion Time In Seconds ........................................................ 54
Table 23: Success Rate Across Tasks .......................................................................................................... 54
Table 24: Unpaired T‐Test Of Success Rate Across Tasks .......................................................................... 55
Table 25: Number Of Errors Across Participants ....................................................................................... 55
Table 26: Unpaired T‐Test Of Task Completion Across Participants .......................................................... 55
VI
Figure Index Figure 1: Sample Screen Of Diamondhelp Application Developed For Collagen (Rich et al, 2005) ........... 10
Figure 2: A Comparison Of Mean System Usability Scale (SUS) Scores By Quartile, Adjective Ratings, And
The Acceptability Of The Overall SUS Score (Bangor et al., 2008) .................................................... 13
Figure 3: Completed Application Demo ..................................................................................................... 20
Figure 4: Prototype One ‐ TV Control Panel .............................................................................................. 21
Figure 5: Prototype One ‐ A/C & Heat Control Panel ................................................................................. 21
Figure 6: Prototype One ‐ Home Lights Control Panel ............................................................................... 22
Figure 7: Prototype One ‐ DVD Player Control Panel ................................................................................. 23
Figure 8: Prototype One ‐ Surround Sound Control Panel ......................................................................... 23
Figure 9: Prototype One ‐ Alarm Clock Control Panel ............................................................................... 24
Figure 10: Prototype One ‐ Home Security Control Panel ......................................................................... 24
Figure 11: Prototype Two ‐ Revised TV Control Panel ............................................................................... 25
Figure 12: Prototype Two ‐ Revised A/C & Heat Control Panel ................................................................. 26
Figure 13: Prototype Two ‐ Revised Home Lights Control Panel ............................................................... 27
Figure 14: Prototype Two ‐ Revised DVD Player Control Panel ................................................................. 28
Figure 15: Prototype Two ‐ Revised Surround Sound Control Panel ......................................................... 28
Figure 16: Prototype Two ‐ Revised Alarm Clock Control Panel ................................................................ 29
Figure 17: Prototype Two ‐ Revised Home Security Control Panel ............................................................ 29
Figure 18: The SUS Survey Used In This Research (In Its Original Form) ................................................... 31
Figure 19: Instructions On How To Use Loop11 ........................................................................................ 34
Figure 20: Average Task Completion Time In Seconds Across Tasks ......................................................... 47
Figure 21: Successful Task Completion Across Participants ...................................................................... 50
Figure 22: Successful Task Completion Across Participants ...................................................................... 51
Figure 23: Mean Ease Of Task Rating, Higher Values Are Better ............................................................... 52
1
Chapter 1: Introduction
Overview
Networked appliances and devices have long held a promise of making it easier for
people to manage their time, lives and energy use. Unfortunately development in this area has
been limited and has found little acceptance in the consumer market outside the hands of a few
knowledgeable power-users. Many consumer electronics companies have attempted to create an
ecosystem of networked and operational home appliances.
The focus of this thesis is on developing a standardized user interface with respect to
functionality and design of total remote control of home appliances and devices by focusing on
three widely-accepted usability measures: effectiveness, efficiency and satisfaction.
Effectiveness and efficiency provide a quantitative measure of success as users interact with the
interface and satisfaction scale provides a qualitative overview of user satisfaction with the
presented system.
Twenty Internet users participated in a remote usability test. Their error rates, satisfaction
and speed of task completion were measured. The three usability measures demonstrated
statistically significant improvements in overall interface design, with a final SUS score of 79%,
the p value for the difference in average time to complete a task between two prototype at p =
.0323, the number of errors across tasks at p = .0206 and number of errors across participants at
p = .0287. This research serves as a basis for building user interface standards by considering end
users goals in terms of functionality users want and expect from their appliances.
2
Background
Motivation
The importance of standardizing user interfaces and considering human factors in product
and system design is impossible to overstate. This was most evident at the Three Miles Island
accident where a series of operator errors led to a catastrophic failure of a nuclear power plant
and a partial meltdown of the nuclear core (Meshkati, 1991). As a consequence of the accident,
the government implemented many changes, including standardization of instruments and
controls for operating the plant (Backgrounder on the Three Mile Island Accident, 2009)
Although standardization has often occurred as a result of accidents or as a preventative
measure to stop accidents before they happen, standardization is happening in other areas that
carry fewer real-world dangers. In areas, such as computer science, the sophistication level of
machines and software available on these machines has happened mainly in part due to these
standardized interfaces (Myers, Hudson & Pausch, 2000). This is most noticeable when
considering user interfaces across many different platforms like Windows, Macintosh and Unix.
A single application such as an Internet browser tends to offer similar functionality and same
functional layout across various platforms. Many of the graphical interface methods were
originally developed by Xerox, and adopted on Macintosh in 1984 (Reimer, 2005) and later
found its way to other platforms such as Windows and Linux. The standardization of Internet
browsers in terms of features, functionality and their look and feel has played a critical role in
shaping the way these technologies have evolved over the past 20 years.
A standardized user interface offers many benefits to end-users. Among these benefits
are: overall increase in user productivity, decrease of stress and frustration, improved
3
interoperability among different devices types such as cell phones to tablets to PC’s and an
improved overall product that is satisfying to use and aesthetically pleasant (Reed, et al., 1999).
Despite of standardizing attempts of UI across personal computers, this standardization
does not translate well onto pocket and portable devices such as smart phones and tablets (Lasky
et al. 1998). Pier and Landay (1992) found that pull-down menus on Windows widgets do not
work well on wall-size displays because they presume a certain height requirement of their users
meaning that not everyone would be able to reach the pull-down menu. In today’s modern
devices which include Apple iPhone, Microsoft Windows Phone 7 and Google Android the UI
across all three major competitors is completely different. Android devices suffer from even
greater fragmentation (Liau Yun Qing, 2010) not only in respect to the number of different
operating system iterations available, but also the variety of non-standard user interfaces
manufacturers are allowed to place on top of the underlying Android operating system.
There have been several promising attempts to bring about a universal interface for
networked home appliances (Rich, et al, 2005). However, their parent corporations have either
canceled these projects prematurely or the research team has moved on to address different
research problems. The reason for a lack of a universal home interface across devices occurs for
several reasons. In many cases it is not feasible for a large corporation to develop a standardized
UI across multiple devices. Companies such as General Electric (GE) develop not only consumer
level products such as phones, microwaves and washing machines but advanced military
equipment and industrial machinery. For a company as large as GE, for instance, it may not be
possible or worthwhile to standardize the UI between a microwave, a washing machine and a jet
fighter. The second reason for a lack of industry-wide standardization across devices is because
4
companies often feel that having their own uniquely identifying UI offers benefits and a
competitive advantage (Rich, et al, 2005).
Developing a consistent and a usable system has long taken a back seat as most
manufacturers focus on price-sensitive consumers and develop their products for the lowest
common denominator, often sacrificing user experience research and design along the way in
order to gain competitive advantage by being the first to deliver a product to the marketplace.
One company that has excelled at maintaining stringent user experience standards across
its devices and has made a business model out of standardized usability is Apple. With the
millions of devices Apple has sold (Apple Inc, 2010), the company has demonstrated that there is
a demand for usability, for consistent user experience, user interface, and that this can be a
highly profitable business model.
The technological march towards faster and more powerful devices has led to increased
complexity. In fact, as consumer devices have grown more complex and rich in features and
functionality, they have in turn become overly complex and difficult to use (Brouwer-Janse et
al., 1992). In response to the growing complexity of these devices researchers have developed
several methods for approaching with this issue.
Simplified UI Methods
This research project is focused on developing user interface standards that would
provide guidance on best layouts and most common features and functionalities of networked
devices. The interface could be loaded on to a tablet device such as an iPad and allow remote
control of an array of networked appliances. At present time, almost all research efforts are
devoted to developing the underlying communication technology while little effort has gone into
5
developing the terminology and a look and feel that remains consistent across multiple devices
such as Android-, iOS- and Windows-based phones and tablet machines.
There are two parts to the current usability crisis (Rich et al, 2005), complexity and
inconsistency. The first part of it is that the proliferation of computers and computer technology
has made it too easy for manufacturers to add features. These features often exceed the capacity
of current UI. The second part is that there is often little to no consistency between devices that
offer similar or same functionality and between the devices coming from the same manufacturer.
Before a user even begins to use a device he or she starts to form a conceptual model of
how a device is going to work (Norman, 2007). Without a conceptual model it becomes difficult
and in some cases impossible to predict the consequences of our actions. This may work well
when the interface is responding a predictable manner; however, as soon as errors are introduced
into the system it breaks down.
The problem with current approaches to home networked appliances and interfaces is that
all projects focus exclusively on the underlying technology and very little attention is paid to the
overall look and feel of the interface. This research explores the underlying set of principles for
home appliances. It attempts to identify user expectations in terms of what they want to see in
their networked appliances in terms of features and functionality, but also provide guidelines for
future developers.
Research Problems
The goals and methods of this research are not the same as those employed by previous
studies. Previous studies looked at networked appliances from a programming point of view.
Other studies focus more on the theory of developing an interface rather than an iterative process
6
of developing an interface. This study, in contrast, focuses on finding out the functionality that
users actually want from their devices, rather than focusing on low-level code that will run below
the surface and not be accessible to users.
Chapter 2: Conceptual Framework
Approaches to Simplified UI
As technology continues to progress at a rapid pace we will find that many of the
products around us will come equipped with computational capabilities and wireless receivers
[Nakajima et al. (2002), Gershenfeld (2000), Gellersen et al. (2000), Kindberg (2000), Hopper
(1999), and Want et al (1995)]. Ultimately, this relentless technological progress will lead to
every device in the world being connected to one another through the Internet (Nakajima et al.,
2002). Devices in the future will not all be a part of a single homogeneous network. Instead they
will run on many different processors and will use different means to provide users with input
and output functionality. It is therefore imperative that this heterogeneity be taken into account
when designing for future interfaces.
In 2005, 43.2 million US households had an Internet broadband connection and 30
million had a personal home network (Wang, 2006, Shehan & Edwards, 2007). Frequently, a
network consists not just of a single device but also of a multitude of devices all interconnected
with one another (Wang, 2006). And although it appears that an average household has an
advanced network set up already and that there aren’t any issues with it, return rates tell a
different story. The return rate for home networking devices is 20-30% and constitutes the most
frequently returned item at large electronics retailers (Scherf, 2002). A user study indicates that
the primary reason for such a large return rate is overall technical complexity in setting up a
7
personal home network (Laszlo et al, 2002). All of these problems can be traced back to the
inception of the Internet. The people who developed the Internet were technically sophisticated
so that the usability and ease of use of networks, networking devices and products was not a
consideration on their minds (Geeksquad, 2010). Research seems to suggest that networked
appliances are suffering from substantial usability crisis and need to be addressed urgently
(Grinter et al., 2005 & Kiesler et al., 2000).
There are several overarching ideas in the area of networked appliances. A vast number
of networked appliances can be classified under one or more of these categories. These big ideas
can be classified into three different approaches to developing smart home appliances. First
approach to smart home development is to simplify consumer electronics interfaces
(Alexandersson, Zimmermann and Bund, 2009; Lieberman and Espinosa, 2006), thereby
allowing each device to be context sensitive and allow for user debugging. Second approach is
often referred to as mixed-initiative (collaborative) interfaces (Rich et al, 2005) whose purpose is
to cooperate with users to satisfy their goals. Third group belongs to the category of goal-
oriented interfaces. This type of device attempts to infer user’s goals based on their actions. The
last method applied to smart home devices is that of self-explanatory interfaces (Lieberman and
Espinosa, 2006). These three approaches to simplifying the UI are documented in the following
sections.
Simplifying Consumer Electronics Interfaces
This first approach of current UI methods can be broken down into infra-red remote
controls, custom installation homes and media centers (Alexandersson, Zimmermann and Bund,
2009). Infra-red remote controls are the most common and well-known interface devices. These
devices are limited in functionality and features because infra-red technology is a one-way
8
communication street and the user can only provide input to the television for instance, but they
cannot receive feedback through the infra-red remote control. The Harmony Remote from
Logitech is only capable of transmitting information one way, however, it is not capable of
providing the user with feedback on things such as what radio station they are tuned into, or
appliance information such as whether the TV is on or off. This makes its functionality limited.
A proposed solution to this has been the development of a universal remote control (URC),
however, the problem with URC’s is that they do not have an ability to control devices beyond
those originally programmed into them because they do not have any means of upgrading.
The custom-made home category of available user interfaces is represented by an array of
appliances that has been specifically designated to provide a total home control to persons with
disabilities and older users. This approach to home control offers substantial benefits to the end
user, however because each setup is custom-tailored to each home the system costs can quickly
become unaffordable to most target users. Multiple projects at MIT’s house_N, Georgia Tech
University’s Aware Home, and others attempt to address the problems of smart home UI.
However, the focus of these projects is on home sensors rather than the UI itself (Lieberman and
Espinosa, 2006).
The third category is a media center category. It is made up of computer-TV hybrids that
extend the computer functionality by delivering user’s media to the TV via the computer. Media
centers such as Windows Media Center pull together the functionality and features of multiple
devices: a TV tuner card, a digital video recorder (DVR), they provide access to on-demand
content, internet streaming video and audio, a surround sound system and a photo viewer. There
are several limitations to this approach. One limitation is that media centers provide little to no
accessibility to persons with disabilities. The second drawback of media centers is that they do
9
not currently offer alternative interface modalities such as gesture control and do not offer a way
to interact with the interface besides using a PC (Alexandersson, Zimmermann and Bund, 2009).
Mixed Initiative (Collaborative) Interfaces
The dominant method of user interaction at the moment is direct manipulation (Rich et al,
2005). Direct manipulation is easy to implement into interfaces that lend themselves to natural
interaction such as pointing, clicking and dragging. As a result of the difficulty in applying these
interaction paradigms into broader Collaborative, or mixed-initiative, interface is based on
Collagen (Rich et al, 2005). Collagen is an “application independent collaboration manager” (pp.
2). This software is placed between the user and the machine and acts as a dialog manager. Prime
example of a Collagen-based interface is DiamondHelp software developed by Mitsubishi (Rich
et al, 2005). DiamondHelp is used for building GUI, it is based on reusable Java framework. This
approach assumes that technology is difficult for humans to use and that only a small fraction of
that functionality. Authors of this study felt that there are two principal reasons why people find
technology difficult to use. Two reasons for this difficulty are because interaction does not
follow any conventions and second reason being inconsistency in user interface design. Authors
of this study note that the system at its present state does not have a standardized UI including
things such as a color pallet, widgets etc. They considered this to be beyond the scope of their
research.
10
Figure 1: Sample Screen Of Diamondhelp Application Developed For Collagen (Rich et al, 2005)
Goal‐Oriented Interfaces
The third approach is that of goal-oriented and commonsense interfaces. These types of
interfaces infer goals from concrete actions and reformulate search queries (Lieberman and
Espinosa, 2006). An example of this would be Roadie. The system uses an OpenMind
Commonsense knowledgebase (Singh, 2002) through a Graphlan (Blum and Furst, 1997)
implementation. Based on the task that a user is attempting to accomplish, the interface will
provide explanations or suggestions for other goals that a user may want to accomplish. For
instance, if a user inserts a DVD into their computer then the interface would provide several
options such as watching a movie on DVD, recording a DVD movie, listening to music on the
DVD etc. (Lieberman and Espinosa, 2006). Roadie can only work with devices that provide a
method for controlling their functions and the devices must be able to query their own state. The
ability of the device to query its own state is what allows Roadie to monitor what is happening
and to interpret that information as user’s actions. They conclude that devices should reduce their
11
own complexity. They should sense, anticipate, and respond to activities at home (Davidoff et al,
2006 & Lieberman and Espinosa, 2006)
Other Interaction Modes and Guiding Principles
The currently dominant human-computer interaction paradigm is direct manipulation
(Shneiderman and Plaisant, 2005). This method of interaction is common to most modern
devices and includes things such as pointing, dragging, sliding etc. The main idea behind direct-
manipulation interfaces is that the objects in a user interface are visible, that the actions can be
undone, the speed with which tasks can be accomplished is fast and instead of using a command
line the user can use direct manipulation through the mouse to move objects and change their
states on the screen (Shneiderman and Plaisant, 2005). These ideas epitomize directly-
manipulate interfaces and have been extended to other forms of interaction such as virtual
reality, augmented reality and tangible user interfaces. According to Shnederman and Plaisant
(2005), direct-manipulation is beginning to emerge as a way manipulate and automate home
appliances.
Donald Norman (2007) outlines seven principles that engineers should adhere to when
developing interaction rules for our machines. The first rule is that designs should be close
approximation of people’s natural interactions and motions they use to interact with one another.
The required input into the machine and the obtained output should be predictable. A designer
needs to provide a good conceptual model as to the workings of the machine. Fourth design rule
is that the output a person gets should be understandable to its users. As part of the natural
interaction a system should be continually aware of itself and it’s own state of being. This
awareness should not hinder its use and should not be so annoying to the user that it is
completely distracting from its primary purpose. The last design rule emphasizes the need for
12
natural mappings in order to aid with the interaction. The rule of natural mappings is a method of
describing a relationship between a control, or an action that a person takes upon the interface
and the way the system reacts. To naturally map something means that the interface corresponds
to similar physical actions and cultural norms the users experience (Norman, p. 23). For
example, turning a steering wheel left logically maps to a car turning left.
System Usability Scale
One of the easiest ways to determine overall usability of a system is by administering a
System Usability Scale (SUS) survey to the participants. The survey provides researchers with an
idea of the overall usability of any given system as a percentage. John Brook developed the SUS
survey (Brooke, 1996). The purpose of this survey, as he put it, was to provide a “quick and dirty
usability scale” that could easily be picked up by any researcher to derive a quick overall
usability rating.
According to Bangor et al. (2008) the success of SUS is attributable to several reasons.
The mean reason, they say, it the SUS survey’s indifference to the platform being tested. The
survey is not tied to any single system, which means it lends itself to use on many different
platforms. The second reason, they state, is the ease of use of the survey. This makes SUS a very
understandable survey to participants and to administrators. Another reason for the proliferation
and success of SUS survey is its ability to provide a single score, a percentage out of one
hundred, which can be easily understood by anyone on the team. Lastly, unlike many other
proprietary and expensive surveys, the SUS survey was made free and available to general
public.
Bangor et al (2008) undertook the most recent empirical evaluation of the SUS survey.
Their study analyzed 2300 SUS surveys over the prior 10 years and their results indicate 85%
13
reliability compared to other usability surveys. To make full sense of results they provided an
overall scoring guide (see Figure 2).
Figure 2: A Comparison Of Mean System Usability Scale (SUS) Scores By Quartile, Adjective Ratings, And The Acceptability Of The Overall SUS Score (Bangor et al., 2008)
The SUS survey is presented to the participants upon completion of their task. The survey
is presented in its original unmodified form. After completing an unmodified SUS survey, the
participants are given a second set of questions soliciting free-form feedback and demographic
information. This information is not critical to the study but it may be useful in discerning any
confounding variables in the study.
Error Rates
Error rates are some of the most important kind of measure in usability studies (Albert,
Tullis, Tedesco, 2010). Errors are defined as a user’s inability to complete a given task or a belief
that they had completed a task when they have not indeed done so. Error rate can be measured in
multiple ways, but it is most frequently a binary measurement. It means that participants either
completed the task or they did not complete a task. Successfully completed tasks are given a one
and unsuccessful or failed tasks are assigned a zero. The software that is used for this remote
usability research does not provide an error rate with the current prototype. This is because the
application is intended for use with web sites rather than applications. The remote usability
14
application, however, does offer the ability to record participant interaction with the testing
prototype. By watching recorded participant sessions the error rates are collected and each task is
marked as complete or incomplete.
This experiment considers two types of error rates: (1) the error rates committed by
individual participants across all eleven tasks and (2) the error rates across all participants for
every given task. This will provide an overall picture of number of errors per participant and
number of errors per task.
Task Completion Times
Task completion times have long been used for usability measurements and are
frequently viewed to be as important as error rates (Albert, Tullis, Tedesco, 2010). Data on task
completion times can be presented in two ways: (1) for all tasks or (2) only the correct tasks.
This study will report both because the information it provides is valuable. Task completion
times are measured by the remote usability testing application Loop11. A task is considered to
have started once a participant loads the application and a task is considered complete once a
participant pushes the “Task Complete” button at the top of the page frame. Time it took each
participant to complete a task is then automatically tabulated and presented.
Chapter 3: Experiment Design
Hypothesis
This section defines the main hypotheses under investigation by this experiment. The
experiment consists three measurements: effectiveness, efficiency, and satisfaction. Data on
15
effectiveness and efficiency are collected through remote usability application provided by
Loop11 service. The last measurement of satisfaction is collected using a SUS score.
Definition
The main goal of this experiment is to build an optimal interface that matches users’
expectations in terms of functionality and usability. This study assesses results according to
effectiveness, efficiency and user satisfaction of the interface. Effectiveness of a design is
defined as the number of errors committed by each participant in every task. Efficiency of a
design is defined as the amount of time it takes each participant to complete a given task. Lastly,
the satisfaction measure is defined as an overall percentage score of user’s satisfaction with the
system.
Performance Data
Three main variables are under investigation by this study, these include: effectiveness,
efficiency and satisfaction. Effectiveness of a design is measured by the number of errors
produced by each design, efficiency measures the average time it takes each participant to
complete a task, and satisfaction is a qualitative measurement intended to gauge overall usability
of a system. A remote usability testing application will collect all data on user interaction. The
hypotheses for each of these variables are:
1. Participants of experiment two should perform fewer errors than participants in
experiment one.
2. Participants of experiment two should perform faster than participants in experiment one.
3. The qualitative measurement of overall system usability satisfaction should be higher in
experiment two than in experiment one.
16
Task Completion and Error Rates
This research considers two types of performance data: average task completion time and
error rates between tasks and participants. Error rates are frequently displayed as binary data and
this research will do the same. The average task completion time is measured in seconds and
milliseconds. The System Usability Scale (SUS) score is used to provide an assessment of the
overall system usability and any improvements that occur through prototyping The SUS score is
reported as a percentage. It is intended to present a quick and easy way of comparing qualitative
results between the two prototypes.
Experiment Setup
Introduction
This research will test two iterations of a prototype. The research will take place online in
an automated remote usability study through a service Loop11. This service allows researchers to
create tasks, administer pre and post task questionnaires and provides measures of response times
for every participant and their task completion or failure rates. The study was conducted as an
online remote usability study due to lack of access to proper usability testing facilities at the
researcher’s abode and the geographical disparity between the main research office. Remote
usability testing offers several benefits when compared to standard live testing. Remote testing
allows researchers to test anywhere, to get a geographic and demographic diversity they could
not in the lab (Bolt & Tulathimutte, 2010). This experiment is approved by and conducted under
supervision of the principal investigator.
17
Initial Survey
An initial survey was developed before any prototyping or experiments took place. The
purpose of this survey was to gauge people’s interest in remote controlled appliances, which
appliances people would like to control remotely and what functionality they are most likely to
wan to want their remote controlled appliances to have.
Twenty-one people participated in the initial survey. The people were recruited through
online social media web sites such as Twitter and Facebook and through Craigslist. The
questionnaire was developed using Google Documents form. It allows researchers to create an
online survey and uses it to tabulate people’s responses to the questionnaire. Participants were
asked to complete a total of 27 questions. The first question listed the most frequently found
appliances in a home. The participants were asked, “What devices or appliances would you like
to be able to control remotely? Check all that apply.” Table 1 summarizes participant responses
to this question.
18
Table 1: Summary Of Participant Interest In Remote Control Of Given Appliances
Device Name 1 TV 2 Air Conditioning 3 Lights 4 DVD Player 5 Heater 6 Surround Sound 7 Alarm Clock 8 Home Security System 9 Door Locks
10 Stereo 11 Clothes Washer 12 Dishwasher 13 Computer 14 Clothes Dryer 15 Stove 16 Microwave 17 Fridge 18 Other
After completing this question all participants were asked about detailed functionality of
each of the devices. Subsequent questions for each of the devices followed the following format:
“Which of the following actions would you most likely want to perform remotely on your (insert
device name here)” The participant was then given a list of the most common functionality for
each of the devices. They could check as many of the features of each device as they wanted.
They could mark none or other and write in their own response as an option also.
In total, twenty-one participants completely filled in the survey. This survey was used as
a basis for creating the two following prototypes. Results of this survey can be found in
Experiment Results chapter of this research under “Initial Survey Results”.
19
Prototype 1 Setup
The first prototype was developed based on initial feedback from the user survey. The
survey solicited input on two main areas: (1) the devices a user is likely to want to control
remotely through a tablet-like interface, and (2) the granularity of control users would like to
have over those devices. Axure Rapid Prototyping (RP) application was used develop a high
fidelity functional application mockup. Axure RP is used by user experience professionals in the
field and allows for rapid prototyping with varying degrees of fidelity. Based on the results from
the initial survey eight devices and their full functionality were implemented for the initial
mockup.
Figure 3 is a screenshot of a completed application demo. The application consists of
seven main devices: television, air conditioning & heat, home lights, DVD player, surround
sound, alarm clock and home security. Although the initial user survey listed Air Conditioning &
Heater as two separate devices, for the purposes of this application the two were combined into
one menu. This was done primarily because the two devices offer very similar or same
functionality and because in many modern homes with an air conditioner, the same control panel
is used to set the cold or heat index in a home.
20
Figure 3: Completed Application Demo
All buttons are laid out on the left-hand navigation and provide an illusion that they are
connected to the main menu display, which is framed by a black border. The two additional
buttons found in the prototype are “Switch User” button and “Add new device…” button. These
buttons provide no functionality in the current demo but were added as placeholders for future
functionality.
Pushing on the button labeled “TV” brings up the TV control panel (see Figure 4). The
TV control panel consists of the on/off component, channel keypad, volume control and buttons
for basic setup. In this application demo all buttons respond to pushes by turning yellow and
timing out after several seconds to denote they have been pushed. Pushing the “Turn On” button
on any of the control panel changes the button label from a red “Off” icon to a green “On” icon
on the device button.
21
Figure 4: Prototype One - TV Control Panel
There are two ways for users to know their current location in the application. The
primary way is the device name inside each control panel. The secondary way is by changing the
button border to a dark black and making the button itself darker. These two visual cues are a
way of telling the user where they’re at within the interface.
The A/C and Heat control panel is simple (see Figure 5). It is made up of an on/off
component and a method for controlling home temperature and humidity. The system also allows
users to select an auto-off time.
Figure 5: Prototype One - A/C & Heat Control Panel
22
The Home Lights control panel is also straightforward (see Figure 6). The panel is broken
up into three sections. There is a global light control referring to “All lights” and it allows users
to turn on or off all lights in a house. Second portion is granular and allows users to see
individual status of lights throughout the house. The last section of the control panel is the “Add
new room…” button, which allows users to add light control to additional areas of their homes.
Figure 6: Prototype One - Home Lights Control Panel
The DVD Player control panel was a significant challenge because users requested total
control of their DVD player through this interface (see Figure 7). Only some functionality of a
DVD player was implemented. This was done due to lack of standards between manufacturers
and different model numbers of DVD players, since each one offered different functionality and
capabilities. A total control of all of DVD Player’s functionality would be difficult to replicate in
an application.
23
Figure 7: Prototype One - DVD Player Control Panel
Surround sound (see Figure 8) was another challenge because of different modes and
functionality each receiver box offers. Two main functionalities, however, are present in all
surround sound receiver boxes. These are: the ability switch various audio and video input
modes instantly and the ability to change volume.
Figure 8: Prototype One - Surround Sound Control Panel
24
Alarm clock control panel (see Figure 9) was straightforward because an alarm needed to
offer only one basic feature and that is the ability for users to select hours and minutes.
Figure 9: Prototype One - Alarm Clock Control Panel
Lastly, Home Security control panel (see Figure 10) is split into three sections like the
lights control panel. There is a global control of a security system, which allows users to arm and
disarm the home security alarm. The second section offers more granular view and control of
locks around the house. In this case controls for Front and Back door are only provided.
Figure 10: Prototype One - Home Security Control Panel
25
Prototype 2 Setup
Based on Experiment 1 and feedback received from users on Prototype 1, a second
Prototype was developed. Second prototype was build around the idea that common functionality
should be grouped into steps and the entire system took on a more wizard-like user interface.
Sections were visual separated and logically grouped, both within each control panel and across
different control panels.
The most obvious immediate difference between Prototype 1 and Prototype 2 is that TV
and Home Lights were left as “On” by default. Jumping into the TV control panel we are
presented with a different user interface (see Figure 11).
Figure 11: Prototype Two - Revised TV Control Panel
The first step to the control panel of every device now is the ability to turn the device on
or off. The reason this functionality was called out in step one is because in the Prototype 1,
users were allowed to click anywhere in the demo and they were not sure how to proceed or what
steps to take and in which order. Often time users would change the channel, turn up volume and
26
lastly turn on the TV itself. Each section was separated by a pleasant blue hue and each section
has a brief header describing what actions to take.
Based on users interaction with Prototype 1 two new components were added: (1) an
output to component and (2) status of device component. The “Output to” component allows
users to select where they would prefer to hear the sound coming from, the TV or the Surround
Sound. This allows them to skip the secondary step of going into the Surround Sound control
panel and having to select and figure out which Audio/Video input corresponds to the television.
A status component was added to all the devices where it made sense. The status component
gives an overview of the current state of a device, in this case the television. It lets the user know
that the TV is on, what the current channel is set to, and that sound is being sent out through TV
speakers rather than surround sound.
For A/C & Heat (see Figure 12) no functionality was added, but the ability for the A/C to
turn off automatically was moved to the right hand side into Step 3.
Figure 12: Prototype Two - Revised A/C & Heat Control Panel
27
Home lights (see Figure 13) benefited from visual separation and by having the dark grey
background of each component changed to white. Compared to Figure 6 each background panel
is now clearer, contrasts well with the background and it is immediately known what
functionality goes with what.
Figure 13: Prototype Two - Revised Home Lights Control Panel
The DVD player control panel also received substantial changes (see Figure 14). Based
on users interaction with Prototype 1, two new components were added: (1) an output to
component and (2) status of device component. The “Output to” component allows users to
select where they would prefer to hear the sound coming from, the TV or the Surround Sound.
This allows them to skip the secondary step of going into the TV control panel. A status
component was also added. The status component gives an overview of the current state of a
device, in this case the DVD Player. It lets the user know that the DVD Player is off, and that
sound is being sent out through TV speakers rather than surround sound.
28
Figure 14: Prototype Two - Revised DVD Player Control Panel
Surround sound (Figure 15) and Alarm Clock (Figure 16) got similar treatment. Surround
sound was made more streamlined by narrowing down the container boxes for all the buttons and
adding a status box. An alarm clock also got a status message box which lets users know whether
the clock is set or not. A “Set” button was also added to the Alarm Clock.
Figure 15: Prototype Two - Revised Surround Sound Control Panel
29
Figure 16: Prototype Two - Revised Alarm Clock Control Panel
Home Security (see Figure 17) again much like the Home Lights section was helped out a
lot by changing the background box color to white which allowed it to contrast more with the
overall application background.
Figure 17: Prototype Two - Revised Home Security Control Panel
That constitutes all the changes made across two prototypes based on user feedback. Next
sections discuss the experimental setup and statistical and qualitative findings.
30
Scoring the System Usability Scale (SUS)
In addition to retrieving quantitative data such as task success rate and response time to
each task, which can be acquired from the remote usability study, a self-reporting survey on the
overall system usability is useful in determining the overall usability of a system. The use of a
self-reporting survey to gauge overall usability of a system is common practice in the usability
field. Bangor, Kortum & Miller (2008) used a SUS form to test its reliability against other
popular usability form and found that SUS is commonly used amongst usability practitioners.
Although quantitative measures of response time and success rates are critical in determining the
usability of a system, the use of a SUS is an important piece of information that allows us to
glean into users’ perceptions and attitudes.
The SUS is used to produce a single number, which is then used to provide researchers
with a composite score of the system. It is important to note that each score on its own is
meaningless. Sum up all the scores first in order to compute the overall score. Each one will
provide a contribution of anywhere between 0 and 4. Questions 1, 3, 5, 7 and 9 are scored by
taking the score and subtracting 1 from each score. For example, if a participant had marked the
second box, the score would be two for that checkmark. A one is then subtracted from that two
and the overall score of one for this question is yielded. Questions number 2, 4, 6, 8 and 10 are
scored by subtracting the position number from number 5. For example, if a participant had
marked the second box, the score would be two for that checkmark. That two is then subtracted
from 5 and it yields an overall score of 3. The sum of all scores is then multiplied by 2.5 and this
produces an overall system usability score. These scores can only range between 0 and 100. The
easiest way to interpret the SUS score is to use a school grading scale (e.g., 0-60 = fail, 60-70 =
D, 70-80 = C, 80-90=B and 90-100=A). (Bangor, Kortum, & Miller, 2008)
31
The following is the System Usability scale used in this research:
Figure 18: The SUS Survey Used In This Research (In Its Original Form)
32
Literature review indicates availability of other usability scales such as After Scenario
Questionnaire (ASQ) developed by IBM, Computer System Usability (CSUQ) developed by
IBM, Usefulness Satisfaction and Ease of Use (USE) scales (Bangor, Kortum, & Miller, 2008).
These scales are considered effective, however, they often require participants to complete as
many as 50 questions. Most subjective measures up until the SUS form were gained through
attitude scales such as CUSI (Kirakowski and Corbett, 1988). However, the problem with this
scale is that is not specific to any given system because they only measure attitudes.
Because this was an online usability study it was important to know that the participants
were actually reading the questions and understanding what is being asked of them. Due to this
concern, a “speed bump” was introduced into the questionnaire. Speed bumps are commonly
used in testing (Rubin, 1994). It is a question that tests reading comprehension. For example, in
this usability test, as the 8th question participants were asked to “For this question please check 1
(strongly disagree)”. Participants who had read the question would check 1, those who
supposedly skimmed through the entire survey gave this question a different value. Participants
who answered anything but 1 for this question were removed from the study.
Experiment Procedure
Twenty people participated in the first and second study. Participants were recruited
through social media networks such as Twitter, Facebook and through web sites like
Craigslist.org. The web prototype was developed using Axure RP Pro 5.5 for Macintosh. Axure
was used to develop a fully functional wireframe along with conditional logic built into the
prototype. The wireframe was designed for optimal display at a resolution of 800x600 pixels.
The font used to display the instructions was the standard Arial font.
33
The experiment was automated and conducted remotely by each participant. Loop11
remote usability test application was used to conduct the experiment. The program allows
researchers to create a self-moderated usability study. The service lets researchers set up remote
usability tests by creating their own tasks and pre and post task questions. Each task is displayed
as a frame bar with two buttons “Abandon Task” and “Task Complete”. On the back end,
Loop11 would track mouse clicks, mouse movements and it would measure the response time to
complete each task as well as the path each participant would take to complete the task. After the
participants completed all the tasks they were then forwarded to a qualitative survey. The
qualitative survey used was a System Usability Scale (SUS) a free likert scale developed by
Digital Equipment Corporation. It is a 10-question simple scale used to determine the overall
system usability.
After a study is created, the program generates a link to the online application or web site
that’s being tested. From this point on all data collection and experiment is automated. After
clicking a link to begin the study each participant was given a brief introduction into the study.
This is a study on the ease of use of consumer electronic devices. We will
show you an unfinished prototype and we would like your feedback. This
study is a research study conducted for and on behalf of Northeastern
University in Boston, MA. If you feel you have questions about this
survey or want to learn more about this study please contact Sanel at
[email protected] or my phone number at xxx-xxx-xxxx. You
will need to set aside 10 to 15 minutes to participate including the post-
study survey. Study consists of 11 tasks. Second part of the study is a
follow-up survey. It should take under 3 minutes to complete.
34
Instructions: Although this application is running on the web, it is
intended to work on a device such as an Apple iPad or a similar tablet
device. You need to use your mouse clicks to indicate your actions.
Because we can’t tell where your mouse is moving you will have to click
on icons and buttons to indicate an action. If you can’t complete the
task, you can use the “Abandon Task” button. Thank you!
Once the participants click the “Begin Evaluation” button, they are given brief
instructions on how to interact with the usability test (Figure 19).
Figure 19: Instructions On How To Use Loop11
35
Task List
All participants were asked to complete the tasks in Table 2: Table 2: List Of Tasks And Order They Were Presented To The User
For each of the 11 tasks a user will be presented with a task he or she needs to complete.
Once the participant gets to a point in the prototype that they feel they are complete with a given
task they push the “Task Complete” button. In case a participant is not sure what to do or how to
complete a task they are told to push the “Abandon Task” button. Once again, before the task
begins the participants are reminded that they are not being evaluated but that the web site is
being evaluated. Once started, participants go through 11 tasks. Upon completion of 11 tasks
participants are thanked and reminded to take the final survey. The survey consists of a SUS
Task Number
Task Description
Task 1 You want to watch your favorite show on channel 7. How would you do this? Task 4 The neighbors are making a lot of noise and you can’t hear your show. Using the mouse
clicks to indicate the steps, show what you would do. Task 8 You are feeling cold and you would like to make it warmer in the house. Using the
mouse clicks to indicate the steps, show how you would do this. Task 6 You are going to sleep soon and you want the heat to automatically turn off in half an
hour. Using the mouse clicks to indicate the steps, show how you would do this. Task 3 You are already in bed but can’t remember if the lights are on in your kitchen. How do
you make sure the lights are off? Task 5 You don’t want to watch the previews at the beginning of a DVD movie. Using the
mouse clicks to indicate the steps, show what you would do. Task 10 You’ve inserted a new disk into the DVD player, and want to watch a movie. Using the
mouse clicks to indicate the steps, show how you would do this. Task 2 You turned on your video game system but you can’t hear or see anything on the TV.
You know the gaming system is in fully functional. What would you do? Task 7 You need to make sure you wake up by 6:45 a.m. tomorrow. Using the mouse clicks to
indicate the steps, show how you would do this. Task 11 You left for vacation and want to make sure the intrusion alarm is set up. Using the
mouse clicks to indicate the steps, show how you would do this. Task 9 You are at work but you need to let someone into your house. They don’t have your
house key but you need to let them in. Using the mouse clicks to indicate the steps, show how you would do this.
36
scale and a demographic form. Overall study takes on average between 10 to 15 minutes to
complete including the post-task survey. This concludes the experimental procedure.
Chapter 4: Experiment Results
The results of this study appear promising. Based on statistical and qualitative results of
the results, making incremental improvements to design results in an improved overall design.
However, further research is needed to increase user satisfaction with the overall system and to
decrease the overall error rate.
Initial Survey Results
Before any of the tests had been started, a 27-question survey was administered to
potential users of the system. The first question listed the most frequently found appliances in a
modern home. The participants were asked, “What devices or appliances would you like to be
able to control remotely? Check all that apply.” Participant responses are broken down in Table
3.
37
Table 3: Breakdown Of Participant Responses To Short Survey
Device Name Number of Responses Percentage of Responses 1 TV 16 76% 2 Air Conditioning 15 71% 3 Lights 15 71% 4 DVD Player 14 67% 5 Heater 13 62% 6 Surround Sound 13 62% 7 Alarm Clock 12 57% 8 Home Security System 11 52% 9 Door Locks 10 48%
10 Stereo 10 48% 11 Clothes Washer 6 29% 12 Dishwasher 6 29% 13 Computer 6 29% 14 Clothes Dryer 5 24% 15 Stove 4 19% 16 Microwave 3 14% 17 Fridge 1 5% 18 Other 1 5%
Based on the number and percentage of responses, only those devices with one third of
responses or more were selected for implementation in current interface. In total, participants
were given 18 different devices to choose from. In order to develop a prototype low-scoring
devices had to be removed first. The cut-off point for making it into the final prototype was
having one third or more responses. Only 9 devices received a rating of 33% of greater.
Although other devices had received a good number of responses they were not select for
the initial prototype. They were not chosen because upon closer analysis it turned out that even
though participants may have said they want to have remote control of a device, when given a
question detailing the available functionality for that device they did not choose any of the
functionality. For example, for a clothes washer 29% of participants said they wanted remote
38
control of it, however 38% of overall participants listed “None of these” when asked what
functionality they would actually want in a clothes washer.
Majority of participants expressed no interest in remote control of a fridge, saying “Some
things require you to move to the object anyways, like a refrigerator. A remote that brought the
food to the table would be nice.” Other reasons for not wanting remote control of a refrigerator
included human comfort or a need that the object can provide, with one participant saying: “One
of the greatest joys of refrigerators is opening it to see what food is inside, then coming back
minutes later as if expecting to find something new. Having a remote that can check fridge
inventory would destroy that.”
A few participants noted security concerns with the appliances and they felt that things
like a stove were too unsafe to operate remotely “One should never be able to turn on products
that can catch fire, etc. without being in the room. Now turning off only the items may be useful,
but never turning on.”
Due to the limitations of the survey tool it was impossible to segregate individual
participant responses from the group. Based on the response summary from Table 3, the devices
selected for prototyping were as follows: TV, A/C, home lights, heater, surround sound, alarm
clock, door locks and a home security system.
Because surround sound and a stereo system offered similar or replicating features and
functionality, they were grouped into a single device under surround sound. The same decision
was made for home security and locks whose grouping makes logical sense.
Deciding which features to prototype was also contingent on survey responses. Table 4
summarizes the responses for remote TV functionality. Not surprisingly, the most commonly
39
requested feature for a remote controlled TV is the ability to turn the TV on and off, followed by
changing the volume and the audio/video input. All the functionality that was requested by more
than forty percent of respondents was implemented in the final design.
Table 4: Most Frequently Requested TV Features And Functionality
Feature or Functionality Number of Responses
Percent of Responses
Turning the TV on and off 17 81.00% Mute the TV 16 76.00% Changing the volume 16 76.00% Changing the video/audio input 14 67.00% Access the TV menu system to change the TV preferences 14 67.00% Turn the closed captioning on and off 9 43.00% Set a favorite TV channel. 9 43.00% View current TV status 8 38.00% Turning secondary audio programming (SAP) on and off 8 38.00% Setting an automatic timer so that the device can turn on based on/off a schedule
7 33.00%
Other 0 0.00% None of these 0 0.00%
Next set of questions refers to the functionality of a heater and an air conditioner. Most
frequently requested features for the heater and the A/C can be found in Tables 5 and 6,
respectively. All features of both the heater and the A/C were implemented for the first
prototype. Only one participant stated that they do not want to have any remote control over their
heater, while none of the participants had the same thoughts about the A/C. Due to the similarity
in functionality of the two devices, these two devices were combined into a single control panel
in the final prototype design. For both the heater and the A/C majority of respondents, 86% for
each, said they want to be able to adjust the temperature remotely. While 67 and 71% of
participants for heater and the A/C, respectively, wanted the ability to adjust the length of time
the device stays turned on. In other words, participants wanted an auto-off switch.
40
Table 5: Summary Of Responses For The A/C
Feature or Functionality Number of Responses
Percent of Responses
Turn the A/C on or off remotely 18 86% Adjust the A/C temperature 18 86% Adjust the length of time the A/C stays on
15 71%
Adjust the A/C humidifier 13 62% None of these 0 0% Other 0 0%
Table 6: Summary Of Responses For Heater
Feature or Functionality Number of Responses
Percent of Responses
Adjust the heater temperature 18 86% Turn the heater on or off remotely 17 81% Adjust the heater humidifier 14 67% Adjust the length of time the heater stays on
14 67%
None of these 1 5% Other 0 0%
The following table, Table 7, refers to the responses received regarding the lights control
in a home. Although certain lights switches and fixtures come with specialized functionality such
as the ability to dim the lights, the basic functionality of a light bulb has stayed the same since
it’s invention. People either want the lights on or off. Forty-three percent of participants
responded by saying they wanted an ability to check the status of lights in each room and this
functionality was built into the prototype.
41
Table 7: Summary Of Responses For Lights
Feature or Functionality Number of Responses
Percent of Responses
Turn the lights on or off 18 86% Check the status of lights in each room 9 43% None of these 1 5% Other 0 0%
DVD player functionality (Table 8) also received a lot of responses. Like all other
devices, participants requested the ability to turn the DVD player on or off remotely as their top
choice. Next they wanted an ability to play a DVD movie or skip and rewind to next chapter
followed by adjusting the DVD player settings. The DVD player settings were not completely
implemented because of variability between manufacturers and within the brands themselves.
Table 8: Summary Of Responses For DVD Player
Feature or Functionality Number of Responses
Percent of Responses
Turn the DVD player on or off remotely 17 81% Play a DVD movie 16 76% Skip or rewind to the next chapter 15 71% Adjust the DVD player settings 10 48% Have total control of the DVD player remotely without using a DVD remote
8 38%
None of these 1 5% Other 1 5%
Surround sound (Table 9) and a stereo system were combined into a single control panel.
Results for the surround sound only are displayed below. Majority of participants requested an
ability to change the volume and the audio/video inputs on the two devices, while a substantial
number of participants also said they would like to view current operational status of the
surround sound.
42
Table 9: Summary Of Responses For Surround Sound
Feature or Functionality Number of Responses
Percent of Responses
Turn the surround sound on or off 15 71% Adjust the volume 14 67% Change the video/audio input of surround sound
13 62%
View status of the surround sound 10 48% None of these 2 10% Other 0 0%
Home security (Table 10) and locks (Table 11) were originally proposed as two separate
devices and two different control panels. However, upon examining the functionality offered by
both devices, it made logical sense to group the functionality of the two devices into one. Home
security systems and locks are often tied together in people’s minds and putting locks together as
part of security seems to have made sense.
Table 10: Summary Of Responses For Home Security
Feature or Functionality Number of Responses
Percent of Responses
Turn the alarm on or off remotely 13 62% Set the alarm remotely 11 52% View alarm settings remotely without being able to change them
8 38%
None of these 5 24% Other 1 5% Table 11: Summary Of Responses For Door Locks
Feature or Functionality Number of Responses
Percent of Responses
View whether the house is locked or unlocked
16 76%
View overall status of home security 16 76% Arm or disarm the home security system remotely
13 62%
Lock or unlock the home remotely 11 52% None of these 3 14% Other 0 0%
43
System Usability Scale (SUS) Results
The experimental results of the post study questionnaire are very positive. All twenty
participants (ten for prototype one and ten for prototype two) completed the System Usability
Scale (SUS) survey and the demographic questionnaire. The questions were presented exactly as
they are shown in the SUS survey (see Figure 18) with the addition of a speed bump question
whose purpose is to detect any users who might simply be skimming through the form and not
reading the questions themselves. It’s important to note that these results do not serve as the only
factor in determining the usability of a system or application. The SUS should be used in
conjunction with other performance data such as average response time and number of errors. On
it’s own, however, the SUS score offers a good indication of the overall usability and any trends.
SUS Results Analysis
First, the answers to the questionnaires were rated. The weight of each question was
determined according to the instructions provided by Brooke (1996). For example, questions
number 1, 3, 5, 7 and 9 are scored by taking the value a participant had marked, which could be
anywhere from 1 to 5 and subtracting 1 from each score. Questions number 2, 4, 6, 8 and 10 are
scored by subtracting the value a participant had marked from a total number of 5. If a
participant had answered “Somewhat Disagree” which has a value of two, then a two would be
subtracted from number five giving a total value of 3 to that answer. The scores for each question
were then tabulated and overall score calculated based on that table.
Table 12 contains the SUS scores for each of the ten participants across all ten questions
for Prototype One only. The participants had to choose between five options on a scale ranging
from “Strongly Disagree” (1) to “Strongly Agree” (5).
44
Table 12: Prototype One - Summary Of Responses To The SUS Survey
Participant q1 q2 q3 q4 q5 q6 q7 q8 q9 q10 SUS Score
1 4 2 4 1 4 1 4 2 4 2 80.0 2 4 3 3 2 3 2 4 2 3 2 65.0 3 3 3 4 1 4 2 3 3 4 4 62.5 4 5 1 5 1 5 1 5 5 5 1 90.0 5 2 1 5 1 5 1 5 1 5 4 85.0 6 3 1 4 1 5 1 5 1 5 1 92.5 7 3 2 4 3 4 4 4 2 2 3 57.5 8 3 2 4 2 3 1 4 2 4 2 72.5 9 2 4 3 1 2 2 4 1 4 4 57.5 10 3 4 2 2 3 3 3 3 3 4 45.0 Overall SUS 70.8
The last column labeled “SUS Score” is an excel formula which takes all the values in a
row and calculates the SUS Score and displays it as a percentage for each of the ten participants
for Prototype One. The last row and column displays an overall SUS score. The SUS Score for
Prototype One was 70.8%, which according to Figure 2 falls in between marginal and good
categories. The lowest SUS score belongs to participant 10 at 45% overall and the highest
recorded SUS score was 92.5% for participant 6. Statistical data are further summarized in Table
13.
Table 13: Prototype One - Summary Of Statistical Data For SUS Survey
Mean SD Median Mode Group 1 Participants 70.8 16.4 68.75 57.5 Experiment two survey was exactly the same as experiment one. The second experiment
consisted of ten participants who were recruited online and at the end of the eleven tasks they
were asked to complete the SUS. Data collected from Prototype Two are summarized in Table
14 below. Table 14 contains the SUS scores for each of the ten participants across all ten
questions for Prototype Two only. Just as in Prototype One, participants were asked to rate the
usability of the system on a scale of (1) “Strongly Disagree” to (2) “Strongly Agree.”
45
Table 14: Prototype Two - Summary Of Responses To The SUS Survey
It is evident from comparing the two tables that data for Prototype Two result in much
larger individual SUS Scores and a greater overall SUS Score. The total overall SUS Score for
Prototype Two is 79.8%. For all the participants in the first experiment, the mean SUS score was
70.8%, which is a total overall increase from Prototype One by 9%. The last row and column
gives an overall SUS score. The lowest SUS score belongs to participant 3 at 57.5 percent overall
and the highest recorded SUS score was 100 for participants 2 and 7. Statistical data are further
summarized in Table 15.
Table 15: Prototype Two - Summary Of Statistical Data For SUS Survey
Mean SD Group 2 Participants 79.8 17.8
For all the participants in the first experiment, the mean SUS score was 70.8 with a
standard deviation of 16.4. The results of the second study based on the feedback and
improvements done to the original design indicate a substantial increase in the overall System
Usability Survey score. A passing mark for any given system is 70% and 79.8% is just .2% shy
of 80% needed to fall into the “excellent” category.
Participant q1 q2 q3 q4 q5 q6 q7 q8 q9 q10 SUS Score 1 4 1 5 1 4 1 5 1 5 1 95.0 2 5 1 5 1 5 1 5 1 5 1 100.0 3 4 3 4 3 3 4 3 2 4 3 57.5 4 2 2 4 1 3 2 4 2 4 1 72.5 5 3 1 4 2 4 2 4 3 4 2 72.5 6 4 1 4 1 4 1 4 1 4 1 87.5 7 5 1 5 1 5 1 5 1 5 1 100.0 8 3 3 4 1 3 2 4 2 4 2 70.0 9 2 3 3 1 2 3 4 5 3 2 50.0 10 4 1 4 1 5 1 5 1 4 1 92.5 Overall SUS 79.8
46
Performance Data Results
General Results
Performance data refers to the data collected by the remote usability application. This
includes participant’s response times (e.g., how long it took them to complete each task) and to
participant’s error rates (e.g., how many participants completed the tasks correctly). These data
allow greater insight into how the tests went, such as what tasks produced greatest error rates and
which tasks had the longest or shortest completion times. Average task completion times will be
considered first. Table 16 provides an overall view of average completion times in seconds.
Table 16: Average Task Completion Time
Completion Time in Seconds Test 1 Test 2 Task 1 51.3 46.30 Task 2 85.1 64.30 Task 3 39.9 27.00 Task 4 54.7 37.70 Task 5 36.7 37.40 Task 6 50.2 31.80 Task 7 40.2 31.50 Task 8 38.0 22.90 Task 9 51.0 26.40 Task 10 29.9 19.00 Task 11 42.1 26.70
This same data is summarized in Figure 20. It is evident from the figure that almost every
task in experiment 2 was completed in less time than the same task in experiment 1. The taller
the bar, the more time it took to complete a task, therefore, shorter is better in this instance. For
Task 5, however, it appears that the completion time was slightly longer than in Experiment 1. In
Experiment 1 task five took 36.7 seconds to complete and in Experiment 2 it took 37.40 seconds
to complete. This could be attributed to the fact that some participants expressed difficulty with
47
completing this task, and although both groups managed to complete the task, the amount of time
it took them to understand the question and find the information was the same.
Figure 20: Average Task Completion Time In Seconds Across Tasks
Task success rate can be viewed in two ways. The first way to look at it is as a percent of
tasks each participant completed successfully. The second way to look at it is as a percent of
participants who completed each task successfully. The percent of tasks each participant
completed successfully can be found in the right column of Table 17. The percent of participants
who completed each task successfully can be found in the last row of Table 17. The remote
usability application can’t provide an automated error table because of the limitations in
technology. Error rates for each participant were derived through participant session video
recordings available through the application.
0 10 20 30 40 50 60 70 80 90
1 2 3 4 5 6 7 8 9 10 11
Second
s
Task
Average Task CompleXon Time (in seconds)
Experiment 1
Experiment 2
48
Table 17: Average Number Of Errors Across Participants And Across Tasks For Experiment 1
Participant Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10 Task 11 Average
P1 0 1 0 0 1 1 1 1 1 1 1 73%
P2 1 0 1 1 1 0 1 0 0 1 1 64%
P3 1 0 1 1 0 1 1 1 0 1 1 73%
P4 1 1 1 1 1 1 1 1 1 1 1 100%
P5 1 1 0 0 1 1 1 1 1 0 1 73%
P6 1 0 1 1 0 1 1 0 1 1 0 64%
P7 1 1 1 1 1 1 1 1 1 1 1 100%
P8 1 0 1 0 0 1 1 1 1 1 1 82%
P9 1 1 1 1 1 1 1 1 1 1 1 100%
P10 0 1 0 1 1 1 1 1 1 1 0 73%
Average 80% 60% 70% 70% 70% 90% 100% 80% 80% 90% 80%
Table 17 and Table 18 are binary tables. A one “1” denotes a successful task completion
and a zero “0” denotes task failure. The total number represents the total number participants
who successfully completed a given task. A total of 8 would then mean that a total of 8 out of 10
participants successfully completed this task.
The lowest-scoring task across all participants was task two “You turned on your video
game system but you can’t hear or see anything on the TV. You know the gaming system is in
fully functional. What would you do?” The can be attributed to not understanding the question
well. Many participants went for the TV functionality and attempted to fix it there. The correct
path would have been to go to surround sound and select a “Gaming” device as the input for the
surround sound, however, instructions did not make it clear that the gaming system and the DVD
player are both connected through the Surround Sound. All participants successfully completed
task seven: “You need to make sure you wake up by 6:45 a.m. tomorrow. Using the mouse clicks
to indicate the steps, show how you would do this.” This was the simplest task to complete out of
all eleven tasks because the path was clear, the participants had to go to the alarm clock panel
49
and it was clear what to do there because of the limited features and functionality of the alarm
clock.
The last column represents the total number of errors produced by each participant across
all tasks. Again these data are binary and one “1” denotes a successful task completion and zero
“0” denotes a task failure. The total number in the right column represents the total number of
successfully completed tasks out of 11. A total of 9 would then mean that a total of 9 out of 11
participants successfully completed this task. The lowest-performing participant between all ten
was participant number two with only 64% of correct answers.
Based on the feedback from experiment one or prototype one, a second prototype was
created and retested with the same test. Second prototype attempts to address shortcomings of
the first prototype. Its primarily goal was to create a more streamlined task scenarios for each of
the components. As a result various “Steps” were added to each of the components and
subcomponents of the main menu were grouped logically so that subcomponents such as “Turn
TV Off” and “On” were pulled out and grouped into step one. This panel in particular was taken
out and given it’s own step because many participants would change the settings for a given
device and turn on the device or appliance after making changes to all the settings. Based on
these observations a second prototype was created and a second experiment was conducted.
The following table (18) presents all the success rates for ten participants in experiment
two or prototype two. In some ways data are similar to experiment one. For example, task two:
“You turned on your video game system but you can’t hear or see anything on the TV. You know
the gaming system is in fully functional. What would you do?” was the lowest scoring task
among all the tasks.
50
Table 18: Average Number Of Errors Across Participants And Across Tasks For Experiment 2
Participant Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10 Task 11 Average
P1 1 1 1 1 1 1 1 1 1 1 1 100%
P2 1 1 1 1 1 1 1 1 1 1 1 100%
P3 1 0 1 0 1 1 1 1 1 1 1 82%
P4 1 0 1 1 1 1 0 1 1 1 1 82%
P5 1 1 1 1 1 1 1 1 1 1 1 100%
P6 1 1 1 1 1 1 1 1 1 1 1 100%
P7 1 1 1 1 0 1 1 1 1 1 1 91%
P8 1 0 1 1 1 1 1 1 1 1 1 91%
P9 1 0 1 1 0 1 1 1 1 1 1 82%
P10 1 1 1 1 1 1 1 1 1 1 1 100%
Average 100% 60% 100% 90% 80% 100% 90% 100% 100% 100% 100%
Figures 21 and 22 offer another view of success rates across tasks and across participants. In
almost all situations, participants completed a greater number of tasks in prototype two than in
one. Figure 22 offers a view across participants and although three participants did better in the
first prototype than in second, overall success rate was greater for prototype two.
Figure 21: Successful Task Completion Across Participants
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11
Num
ber o
f ParXcipan
ts
Task Number
Success Rate Across Tasks between two prototypes
Prototype 1
Prototype 2
51
Figure 22: Successful Task Completion Across Participants
Table 19: Experiment 1 – Mean Task Ease Of Use Ratings
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10 Task 11 Mean
p4 2 2 3 2 2 2 2 2 2 2 2 2.09 p5 3 2 2 2 3 1 2 2 1 2 2 2.00 p6 2 0 2 2 1 2 2 2 2 2 2 1.73 p7 3 2 3 1 2 3 3 3 3 3 3 2.64 p8 3 2 3 2 2 3 3 3 3 3 3 2.73 p10 2 2 2 2 2 2 2 2 2 3 2 2.09 p12 3 2 2 2 0 3 3 3 2 2 2 2.18 p13 2 0 3 2 1 1 3 2 2 2 2 1.82 p15 2 1 2 2 0 1 2 2 1 0 1 1.27 p16 0 1 3 2 2 3 3 3 3 3 2 2.27 Mean 2.2 1.4 2.5 1.9 1.5 2.1 2.5 2.4 2.1 2.2 2.1 90% CL 0.48 0.44 0.27 0.16 0.51 0.46 0.27 0.27 0.38 0.48 0.30
Table 20 tabulates data on participants task ease of rating along with means across tasks and
across participants for Experiment 2.
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10
Task Com
pleX
on Rate
ParXcipant Number
Success Rate Across ParXcipants between two prototypes
Prototype 1
Prototype 2
52
Table 20: Experiment 2 – Mean Task Ease Of Use Ratings
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10 Task 11 Means
p1 3 2 3 2 2 2 3 3 3 3 2 2.5 p2 3 2 3 2 3 3 3 3 3 3 3 2.8 p3 2 1 2 2 1 2 3 3 2 2 2 1.9 p4 3 1 3 1 2 1 2 3 2 2 3 2.1 p5 2 2 3 2 2 3 2 2 2 3 3 2.4 p6 2 3 2 2 2 2 2 2 2 2 2 2.0 p7 3 1 3 2 0 3 3 3 3 3 3 2.5 p8 2 1 3 2 1 2 2 3 3 2 3 2.1 p9 2 1 2 2 0 2 2 2 3 2 2 1.8
p10 3 2 3 2 2 3 3 1 3 3 3 2.5 Means 2.5 1.6 2.7 1.9 1.5 2.3 2.5 2.5 2.6 2.5 2.6 90% ci 0.27 0.43 0.25 0.16 0.51 0.35 0.27 0.37 0.27 0.27 0.27
Figure 24 illustrates the differences between the two prototypes on the mean ease of use task
ratings.
Figure 23: Mean Ease Of Task Rating, Higher Values Are Better
0
0.5
1
1.5
2
2.5
3
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10 Task 11
Mean Ease Task RaXng (0‐3, Higher = Be^er)
Prototype 2
Prototype 1
53
Statistical Analysis
It is important to cover statistical analysis of all tasks and ratings. T-test statistic was used
to analyze the number of errors across tasks, participants and average task completion times. T-
test is used instead of an Analysis of Variance (ANOVA) because ANOVA should only be used
when testing 3 or more groups. These experiments revolve around comparing two different design
prototypes and therefore, t-test was deemed the most appropriate.
Overall, all differences were found to be statistically significant. This includes the task
completion time in seconds (P = 0.0323) (Table 22), number of errors across tasks (P = .0206 )
(Table 24) and number of errors across participants (P = .0287) (Table 26)
All t-tests completed are two-tailed tests. The first t-test was done to compare the average
time it took each participant to complete all the tasks. Data on completion time was collected
through Loop11. A task begins once a page is loaded and ends when a participant pushes the
“Task Complete” button. The numbers in Table 19 are a cumulative average of all ten participants
across both prototypes.
Table 21: Average Number Of Seconds To Complete A Task
Prototype 1 Prototype 2 Task 1 51.3 46.30 Task 2 85.1 64.30 Task 3 39.9 27.00 Task 4 54.7 37.70 Task 5 36.7 37.40 Task 6 50.2 31.80 Task 7 40.2 31.50 Task 8 38.0 22.90 Task 9 51.0 26.40 Task 10 29.9 19.00 Task 11 42.1 26.70
54
Table 20 offers a statistical summary of all the data gained from the table above. The
significance level of an unpaired t-test for data is less than the necessary p < .05 level. The
differences in task completion time are, in fact, significant at p = 0.0323 level, far below the level
necessary to constitute statistical significance. This indicates that the two mean values are indeed
truly different from one another.
Table 22: Unpaired T-Test Of Average Completion Time In Seconds
Group Prototype 1 Prototype 2 Mean 47.1909 33.7273 SD 14.681 12.701 N 11 11 P-value P = 0.0323
The second t-test was done to compare the number of participants who managed to
successfully complete each task between the two prototypes. A higher number in this instance is
better because it means a greater number of participants completed a given task. Ten is a
maximum number. As noted in the earlier analysis data on successful completion of each task
was collected by observing video playback of each participant’s study and denoting his or her
success with a 1 or failure with a 0. The numbers in Table 23 represent a total number of
participants (out of ten) who completed each task on each of the two runs.
Table 23: Success Rate Across Tasks
Prototype 1 Prototype 2 Task 1 8 10 Task 2 6 6 Task 3 7 10 Task 4 7 9 Task 5 8 8 Task 6 9 10 Task 7 10 9 Task 8 8 10 Task 9 8 10 Task 10 9 10 Task 11 8 10
55
Table 24 offers a statistical summary of all the data gained from the table above. The
significance level of an unpaired t-test for data in Table 22 is less than the necessary p < .05
level. The differences in task completion time are, in fact, significant at p = 0.0206 level, far
below the level necessary to constitute statistical significance. This indicates that the two mean
values are indeed truly different from one another.
Table 24: Unpaired T-Test Of Success Rate Across Tasks
Group Prototype 1 Prototype 2 Mean 8.00 9.27 SD 1.10 1.27 N 11 11 P-value P = .0206
Table 25: Number Of Errors Across Participants
Table 26: Unpaired T-Test Of Task Completion Across Participants
Group Prototype 1 Prototype 2 Mean 8.80 10.20 SD 1.62 0.92 N 10 10 P-value P = .0287
Combined, these results are very positive. The significance level of each of these tests
means that there were valuable and valid improvements made to prototype one which resulted in
Prototype 1 Prototype 2
p1 8 11
p2 7 11
p3 8 9
p4 11 9
p5 8 11
p6 7 11
p7 11 10
p8 9 10
p9 11 9
p10 8 11
56
a better overall user experience in prototype two or experiment two. Research indicates that
participants in experiment two outperformed participants in experiment one in almost all cases.
The significance level for this finding is p = .0323. With regards to task success rates, these two
measures were also statistically significant. A comparison of tasks completed successfully by
each participant yields a statistically significant results of p = .0206. A comparison of
participants who completed each task successfully yields a statistically significant result of p =
.0287.
Overall Summary
As documented in the discussion, Tables 22, 24, and 26, summarize the results of the
study. Based on the statistical analysis of differences between the two groups the results of the
study are significant and point to an overall improvement from the original design. There are
some unknowns that warrant further study and examination. For example, participants who
tested prototype one did better than participants in prototype two for task five. This difference
however is only several milliseconds so it could likely be ascribed to random error.
Chapter 5: Conclusion
The number of tablet devices such as iPads sold worldwide has grown to 4.19 (Apple Inc,
2010) million and with the introduction of Android platform on tablet devices the potential
market opportunity for a device that combines total home control in one interface is even greater.
The ability of these devices to remain friendly to abled and disabled consumers is critical.
This research discussed two different incremental design improvements, error rates, task
completion time and an overall System Usability Survey score. Across all measures the results
yielded statistically significant findings. An almost ten point swing in the overall System
57
Usability Scale (SUS) survey puts prototype two on the border of an excellent category. This
means that with further adjustments and testing a prototype could be made to match user’s
expectations much better. With respect to the overall results on mean completion time, statistical
analysis indicates that these results are indeed significant at the p<.05 level, and the same holds
true for statistical comparison across tasks and across participants.
This study is a move in positive direction in opening up the discussion about what
common functionality all devices should share and what is the best method for portraying that
functionality to users. The initial survey requesting participant responses did well in gauging user
interest in different appliance functionality. The survey helped identify some critical features that
users want from their devices and shed light on some features which are generally considered to
be wanted by users but the users never expressed any want or need for them. In a future study the
list of tasks and scenarios should be further considered before testing and implementation. A
future study should consist of a more statistically representative sample of general population.
There are many benefits to running a between subjects study chief of which is easier recruitment
pool for a second follow-up study, however it would be interesting to re-run the study as a within
subjects design rather than a between subjects design. Although an online or remote usability
study has proved useful in this situation, more valuable insight could be gleaned from live in-
person usability studies.
58
Appendix
Appendix A: SUS Survey
59
Appendix B: Instructions to Participants
This is a study on the ease of use of consumer electronic devices. We will show you an
unfinished prototype and we would like your feedback. This study is a research study conducted
for and on behalf of Northeastern University in Boston, MA. If you feel you have questions
about this survey or want to learn more about this study please contact Sanel at
[email protected] or my phone number at xxx-xxx-xxxx. You will need to set aside 10
to 15 minutes to participate including the post-study survey. Study consists of 11 tasks. Second
part of the study is a follow-up survey. It should take under 3 minutes to complete.
60
References
1. Alexandersson, J., Zimmermann, G., and Bund, J. (2009). User Interfaces for AAL: How
Can I Satisfy All Users? In Proceedings of Ambient Assisted Living - AAL. 2. Deutscher
Kongress mit Ausstellung/ Technologien - Anwendungen - Management, pp. 5-9.
2. Apple Inc (October 18, 2010). “Record Mac, iPhone and iPad Sales Highest Revenue and
Earnings Ever.” Press Release. Retrieved November 18, 2010.
http://www.apple.com/pr/library/2010/10/18results.html.
3. Albert, Tullis, Tedesco (2010) Beyond the Usability Lab, p. 131.
4. Backgrounder on the Three Mile Island Accident. (2009). http://www.nrc.gov/reading-
rm/doc-collections/fact-sheets/3mile-isle.html#sources.
5. Bangor, A., Kortum, P. and Miller, J.A. (2008) The System Usability Scale (SUS): An
Empirical Evaluation. International Journal of Human-Computer Interaction, pp. 114-
123.
6. Blum, A., Furst, M. (1997). Fast Planning Through Planning Graph Analysis. Proc. Of
the 14th International Joint Conference on Artificial Intelligence (IJCAI 1997), Montreal,
Canada, pp. 1636-1642.
7. Bolt, N., & Tulatimutte, T. (2010). Remote Research: Real users, real time, real research,
p. 35.
8. Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas,
B. A.
61
9. Davidoff, S., Lee, M.K., Yiu, C., Zimmernan, J., & Dey, A.K. (2006) Principles of smart
home control, to appear in Proceedings of Ubicomp, 2006.
10. Geeksquad, http://www.geeksquad.com.
11. Gershenfeld, N. (2000). When things start to think. Owl Books.
12. Gellersen, H. W., Schmidt, A., & Beigl, M. (2000). Adding some smartness to devices
and everyday thngs. In Proceedings of the third workshop on mobile computing system
and applications.
13. Grinter, R.E., Ducheneaut, N., Edwards, W.K., Newman, M. (2005). The Work To Make
The Home Network Work. In: Proc. Ninth European Conference on Computer-Supported
Cooperative Work (ECSCW 05), pp. 469–488.
14. Hopper, A. (1999). Sentient computing. The royal society Clifford Paterson lecture.
15. Kiesler, S., Lundmark, V., Zdaniuk, B., Kraut, R.E. (2000). Troubles with the Internet:
The dynamics of help at home. Human Computer Interaction, pp. 323–351.
16. Kindberg, T. (2000). People, places, things: Web presence for the real world. In
Proceedings of the third workshop on mobile computing system and applications.
17. Kirakowsi, J, and Corbett, M. (1988), Measuring User Satisfaction, in D M Jones and R
Winder (Eds.) People and Computers IV. Cambridge: Cambridge University Press.
18. Lasky, M. S., McCracken, H., and Bielski, V. (1998). Just how helpful are handhelds?
CNN Interact.
19. Laszlo, J., et al. (2002).V. Home Networking: Seizing Near-Term Opportunities to
Extend Connectivity to Every Room. Jupiter Research.
62
20. Liau Yun Qing. (2010). http://www.zdnetasia.com/enterprises-face-device-
fragmentation-challenge-62204766.htm.
21. Lieberman, H., Liu, H., Singh, P., and Barry, B. (2005). Beating Common Sense into
interactive Applications. AI Magazine 25(4): Winter 2005, pp. 63-76.
22. Lieberman, H., and Espinosa, J. (2006). A goal-oriented interface to consumer electronics
using planning and commonsense reasoning. In IUI ’06: Proceedings of the 11th
international conference on Intelligent user interfaces, pp. 226–233, New York, NY,
USA. ACM Press.
23. Meshkati, N. (1991). Human factors in large-scale technological systems' accidents:
Three Miles Island, Bhopal, and Chernobyl. Industrial Crisis Quarterly, pp. 133-154.
24. Myers, B., Hudson S., and Pausch, R. (2000). Past, present, and future of user interface
software tools. ACM Trans. Computer-Human Interaction, pp. 3-28.
25. Nakajima, T., Ishikawa, H., & Tokunaga, E. (2002). Technical Challenges for Building
Internet-Scale Ubiquitous Computing. In Proceedings of International Workshop on
Object-Oriented Real-Time Dependable Systems.
26. Norman, D. (2007). The design of future things. Basic Books. pp. 152, 23.
27. Pier, K. and Landay, J. (1992). Issues for location-independent interfaces. Tech. Rep.
ISTL92-4. Xerox PARC, Palo Alto, CA. Available at
http://www.cs.berkeley.edu/˜landay/research/publications/LII.ps.
28. Raimer, J. (2005). A history of the GUI. Arstechnica.com. Retrieved on November 30,
2010.
63
29. Raskin, J. (2000). The Humane Interface: New Directions for Designing Interactive
Systems. Addison-Wesley Professional, First Edition, 2000.
30. Reed, P., Holdaway, K., Isensee, S., Buie, E., Fox, J., Williams, J., Lund, A. (1999). User
Interface Guidelines and Standards: Progress, Issues, and Prospects. Interacting with
Computers, pp. 119–142.
31. Rich, C., Sidner, C., Lesh, N., Garland, A., Booth, S., and Chimani, M. (2005).
DiamondHelp: A Graphical User Interface Framework for Human-Computer
Collaboration, in Proc. of IEEE International Conference on Distributed Computing
Systems Workshops, pp. 514-519.
32. Rich, C. (2009). Building task-based user interfaces with ANSI/CEA-2018. Computer,
pp. 20–27.
33. Rich, C., & Sidner, C. (1998). Collagen: A Collaboration Manager Software Interface
Agents. Journal of User Modeling and User-Adapted Interaction, pp. 315-350.
34. Rubin, J. (1994). Handbook of Usability Testing: How to plan design and conduct
effective tests. Wiley, p. 124.
35. Scherf, K. (2002). Consumer Electronics Association (CEA) Conference, Parks
Associates Panel on Home Networking.
36. Shehan, E., Edwards, W.K. (2007). Home Networking and HCI: What Hath God
Wrought? In: Proceedings of Conference on Human Factors in Computing Systems, pp.
547–556.
64
37. Singh, P. (2002). The public acquisition of Commonsense knowledge. Proc. AAAI
Spring Symposium on Acquiring (and Using) Linguistic (and World) Knowledge for
Information Access.
38. Shneiderman, B., and Plaisant, C. (2005). Designing the User Interface: Strategies for
Effective Human-Computer Interaction. Addison-Wesley, Reading, MA.
39. Wang, H. (2006). Networks in the Home: Connected Consumer Electronics. Parks
Associates.
40. Want, R., Schilit, B., Adams, N., Gold, R., Petersen, K., Ellis, J., Goldberg, D., &
Weiser, M. (1995). The ParcTab ubiquitous computing experiment. Technical report
CSL-95-1, Xerox Palo Alto Research Center.