Post on 12-Jan-2016
The Challenges and Potential of End-User Gesture Customization
Uran Oh1 and Leah Findlater2
1 Department of Computer Science2 College of Information StudiesUniversity of Maryland, College Park
�
uranoh@cs.umd.edu | leahkf@umd.edu
Touchscreen gestures are widely used…Who designs these gestures?Design experts.
Apple’s touchpad gestures
1) Tools for supporting designers (developers)to create gestures with ease
Previous Research:
A figure from Gesture Coder
MAGIC:[Ashbrook et al. 2010]
Proton++:[Kin et al. 2012]
Gesture Coder:[Lü et al. 2012]
2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users
[Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011]
Previous Research:
A figure from [Wobbrock et al. 2009]
(2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users
[Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011]
Previous research:
A figure from [Wobbrock et al. 2009]
Our focus: Supporting end-users
Personal gestures for a single user
(2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users
[Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011]
Previous research:
A figure from [Wobbrock et al. 2009]
Our focus: Supporting end-users
Personal gestures for a single userWhy?
MemorabilityEfficiency
Accessibility
Potential Advantages of Self-defined Gestures…
[Nacenta et al. 2013]
Memorability
�
Self-defined gestures improve memorability over predefined gestures
[Ouyang et al. 2012]
Efficiency
�
Gestural shortcuts can be used as an efficient mean of accessing information
Accessibility
[Anthony et al. 2013]
�
Customized gestures may improve accessibilityfor people with physical disabilities
Our Goal: To investigate the feasibility of end-user gesture creation
�
Our Goal: To investigate the feasibility of end-user gesture creation
�
How do typical users create gestures?
What are the challenges therein?
How can we support the process?
v
Task 1 Task 2 Task 3Task 2
Open-EndedGesture Creation
Action-SpecificGesture Creation
Saliency ofGesture Features
Study With Three Tasks
Controlled lab study - 20 participants (age from 20 to 35,
M=29.3)- Prior experience with touchscreen devices- Single one-hour session with 3 tasks- Think-aloud protocol
Study Method
�
Apparatus- Samsung Galaxy Tab
2 (10.1’’ running Android 4.0.4)
v
Task 1
Task 2 Task 3Task 2
Open-EndedGesture Creation
Action-SpecificGesture Creation
Saliency ofGesture Features
Q. Are users able to create new gestures easily?If not, what are the barriers?
�
Task 1: Open-ended Gesture Creation
“Create as many gestures as possible”
�
“Create as many gestures as possible”
Task 1: Open-ended Gesture Creation
• For any purpose
• Any number of strokes, fingers, hands
• As long as they are:easy to draw, easy to remember,
distinguishable
�
Task 1: Open-ended Gesture Creation
“Create as many gestures as possible”
12.2 gestures created on average
(SD = 8.1, range of 5 to 36)
Gestures Created
p3
Total number of gestures and the number of arbitrary gestures are correlated
(Pearson’s r=.47, p=.037)
Task 1: Open-ended Gesture Creation
p5
12.2 gestures created on average
(SD = 8.1, range of 5 to 36)
Gestures Created
p3 p5
Total number of gestures and the number of arbitrary gestures are correlated
(Pearson’s r=.47, p=.037)
Task 1: Open-ended Gesture Creation
p5
Tendency to focus on the familiar“I just thought of gestures my tablet PC had.” (P1)“These gestures are all I use, I cannot be more creative” (P8)
Difficulties Creating Gestures
�
Opaque nature of gesture recognizer“Can I use all fingers?” (P2)
Task 1: Open-ended Gesture Creation
v
Task 1
Task 2 Task 3Task 2
Open-EndedGesture Creation
Action-SpecificGesture Creation
Saliency ofGesture Features
A. Users felt difficulty in creating new gesturesBetter understanding of recognizer is needed
23
Task 1
Task 2
Task 3
Task 2
Open-EndedGesture Creation
Action-SpecificGesture Creation
Saliency ofGesture Features
Q. What is a “good gesture” to end-users?How is it different from recognizer’s perspective?
Task 2: Action-Specific Gesture Creation
Brainstorm gesturesper action
12 Specific ActionsZoom-inZoom-outRotateCopyCutPasteSelect-singleSelect-multiplePreviousNextCall-MomLaunch a web-browser
12 Specific ActionsZoom-inZoom-outRotateCopyCutPasteSelect-singleSelect-multiplePreviousNextCall-MomLaunch a web-browser
Task 2: Action-Specific Gesture Creation
Brainstorm gesturesper action
Task 2: Action-Specific Gesture Creation
Compose custom set of gestures, one per
action
Brainstorm gesturesper action
Task 2: Action-Specific Gesture Creation
Compose custom set of gestures, one per
action
Brainstorm gesturesper action
Task 2: Action-Specific Gesture Creation
Compose custom set of gestures, one per
action
Brainstorm gesturesper action
Task 2: Action-Specific Gesture Creation
Brainstorm gesturesper action
Compose custom set of gestures, one per
action
Create training examples
(4 per selected gesture)
Task 2: Action-Specific Gesture Creation
Brainstorm gesturesper action
Compose custom set of gestures, one per
action
Create training examples
(4 per selected gesture)
Rate satisfaction with the custom gesture
set
Brainstorm gesturesper action
Compose custom set of gestures, one per
action
Create training examples
(4 per selected gesture)
Rate satisfaction with the custom gesture
set
Test recognition accuracy with $N
recognizer
Initial example
Training examples
Task 2: Action-Specific Gesture Creation
Generally Preferred
Accurate
Familiar
Simple/Easy
Intuitive/Natural/Obvious
0 5 10 15 20 25 30
11.34
12.18
15.55
22.69
27.73
Percentage of Gestures (%)
Reasons for selecting a gesture for custom set
Others reasons: Generally preferred, fast, consistent, easy to remember, etc.
Task 2: Action-Specific Gesture Creation
Need for improvement
Participants gave up the opportunity to edit their gesture set to make improvements
Task 2: Action-Specific Gesture Creation
Only two participants were fully satisfied
( M=5.3, SD = 1.1 where 1=negative, 7=positive)
Inability to improve gesture sets
Low Recognition Potential of the Custom Sets$N recognizer (default setting) with 5-fold cross validation
1 2 3 40.7
0.8
0.9
Number of Training Examples
Re
cog
nit
ion
Ac-
cura
cy
�
76–88% accuracy depending on amount of training
Task 2: Action-Specific Gesture Creation
35
Task 2
Task 3
Task 2
Action-SpecificGesture Creation
Saliency ofGesture Features
Customized set can be improved for both user’s and recognizer’s perspectiveA.
Task 1
Open-EndedGesture Creation
36
Task 2
Task 3
Task 2
Action-SpecificGesture Creation
Saliency ofGesture Features
What features do users rely on to distinguish between gestures?Q.
Task 1
Open-EndedGesture Creation
Gesture Features Judged
Orientation
Very slow Very fast slow fast moderate
Scale
Aspect Ratio
Speed
Task 3: Saliency of Gesture Features
Curviness
Pattern Repetition
6 features from Rubine’s recognizer [Rubine. 1991]
Gesture Features Judged
Orientation
Scale
Aspect Ratio
Task 3: Saliency of Gesture Features
Curviness
Pattern Repetition
Finger Count
Stroke Count
Stroke Order
3 touchscreen features
6 features from Rubine’s recognizer [Rubine. 1991]
Very slow Very fast slow fast moderate
Speed
Orientation
Scale
Aspect Ratio
Task 3: Saliency of Gesture Features
Curviness
Pattern Repetition
Finger Count
Stroke Count
Stroke Order
3 touchscreen features
6 features from Rubine’s recognizer [Rubine. 1991]
“Rank the distinguishability of 9 features”
Very slow Very fast slow fast moderate
Speed
Objective features are more distinguishableFeatures that can be consistently interpreted/manipulated are considered distinguishable
“Even if the same person is performing the gesture, it might not have the same speed and size” (P7)
More distinctive
Very fast
Speed
Scale
Pattern
Repetit
ion
Aspect
Ratio
Curvin
ess
Orienta
tion
Stroke
Ord
er
Stroke
count
Finger c
ount
Task 3: Saliency of Gesture Features
Objective features are more distinguishableFeatures that can be consistently interpreted/manipulated are considered distinguishable
“Even if the same person is performing the gesture, it might not have the same speed and size” (P7)
More distinctive
Very fast
Speed
Scale
Pattern
Repetit
ion
Aspect
Ratio
Curvin
ess
Orienta
tion
Stroke
Ord
er
Stroke
count
Finger c
ount
Task 3: Saliency of Gesture Features
Objective features are more distinguishableFeatures that can be consistently interpreted/manipulated are considered distinguishable
“Even if the same person is performing the gesture, it might not have the same speed and size” (P7)
More distinctive
Very fast
Speed
Scale
Pattern
Repetit
ion
Aspect
Ratio
Curvin
ess
Orienta
tion
Stroke
Ord
er
Stroke
count
Finger c
ount
Task 3: Saliency of Gesture Features
43
Task 2
Task 3
Task 2
Action-SpecificGesture Creation
Saliency ofGesture Features
A.
Task 1
Open-EndedGesture Creation
Number of fingers/strokes, stroke order aredistinguishable than speed or size
Summary
Creating new gestures is hard for end-users• Tendency to focus on the familiar• Opaque nature of gesture recognizer
�
Objective features are more distinguishable• Finger/stroke count, stroke order are
more distinguishable than speed and scale
Quality of gesture sets can be improved• Users are not fully satisfied with their
gesture sets• Low recognition potential
Memorability
Efficiency
Accessibility
�
Potential Benefits of Allowing End-User Customization
Take-away Message
Systematic Support is Needed for End-User
Customization
Future Work
Mixed-initiative support for customization
Feedback
EditsTrain
System Gesture set User
47
The Challenges and Potential of End-User Gesture Customization
Uran Oh1 and Leah Findlater2
1 Department of Computer Science2 College of Information StudiesUniversity of Maryland, College Park
�
uranoh@cs.umd.edu | leahkf@umd.edu
Thank you for listening
Questions?