Transcript of 2-Thumb Gesture
2-Thumb Gesture: The Design & Evaluation of a Non-Sequential
Bi-manual Gesture Based Text Input Technique for Touch
Tablets
Khai N. Truong1, Sen Hirano2, Gillian R. Hayes2, Karyn
Moffatt3
1Department of Computer Science
khai@cs.toronto.edu
University of California, Irvine Irvine, CA 92697
{shirano,gillianrh}@ics.uci.edu
karyn.moffatt@mcgill.ca
ABSTRACT We present 2-Thumb Gesture (2TG), a non-sequential
bi-manual gesture-based text input technique for touch tablets,
which enables the user to enter text with both hands by using each
thumb to draw small strokes over the keys on their respective sides
of the keyboard without waiting for their turn in the letter
sequence of a word. The results of a study comparing 2TG to Swype
(a 1-finger word drawing method), suggest that the learning and use
of the 2TG technique to perform text input is comparable with the
commercial Swype technique by those who had no prior experience
with either. Furthermore, participants were able to hold and use
the tablet with both hands without experiencing the substantial
fatigue that results from a one-handed approach. Only 60 minutes
after being introduced to the technique, participants were able to
use the 2- Thumb Gesture keyboard to enter text at 24.43 wpm, with
an uncorrected error rate of 0.65%.
Categories and Subject Descriptors H5.2 [Information interfaces and
presentation]: User Interfaces. – Input devices and
strategies.
General Terms Design, Experimentation, Human Factors
Keywords Text entry, soft keyboard, gesture, bi-manual
interaction
1. INTRODUCTION As mobile devices have grown in popularity, an
increasing number of text-input techniques have been introduced.
One such technique is shape writing—a method that allows the user
to input text by drawing a stroke with a near unique shape
containing points close to all letters found sequentially in a
word. Drawing shapes enables faster input than traditional typing
techniques that require the individual pressing of each letter in a
word. As a result, it is now found commercially in a variety of
products, including ShapeWriter [36], Swype [28], and SlideIT
[26].
More recently, sales of large mobile touchscreen devices (i.e.,
those with screens four inches and larger) have increased. The size
and weight of these devices make them hard to hold steady with only
one hand while interacting with the other for long tasks,
especially text input. As a result, commercial products such as
Keymonk [10] and research efforts such as Bi et al.’s investigation
of bimanual gesture keyboards [2] have begun to explore how words
can be drawn on mobile touchscreen devices using both hands. In
contrast to previous shape writing techniques, which support text
input as the drawing of a single stroke containing points near the
keys found sequentially in a word, these methods break the input of
a word into
two smaller strokes that can be drawn by both thumbs over the keys
sequentially on their respective sides of the keyboard.
In this paper, we demonstrate that the two-stroke gestures for
entering English words are highly unique and hence each stroke can
be drawn over the letters non-sequentially by the two thumbs. We
describe our design and evaluation of a non-sequential bi-manual
gesture based text input technique for touch tablets, called
2-Thumb Gesture or 2TG (see Figure 1). Through a controlled
experiment, we establish the feasibility of learning and using the
2TG technique to perform text input alongside the commercial Swype
keyboard (a 1-finger word drawing software). As expected, those
with prior knowledge of Swype were faster using Swype than 2TG;
however, their learning and performance rates with our two-thumb
version were not statistically different from those with no prior
Swype knowledge. Furthermore, by the end of the study, participants
had begun to learn the gestures for many short words and were able
to perform almost one fifth of the input by simultaneously
gesturing with both thumbs and touching keys out of order with
respect to their turn in the actual letter sequence. These results
suggest that as participants begin to remember more gestures over
time, 2TG
Figure 1. The 2-Thumb Gesture keyboard. The user enters text by
using both thumbs to perform drag gestures across the letters in a
word on their respective half of the keyboard.
can potentially extend similar performance benefits found in the
traditional Swype technique to large-size touchscreen tablets,
while enabling the user to hold and use the tablet with both hands
to eliminate the substantial fatigue that results from a one handed
approach.
2. RELATED WORK The 2-Thumb Gesture keyboard was designed to allow
the user to hold the tablet with both hands and draw two short
stroke gestures with both thumbs simultaneously, touching keys
non-sequentially. Bi-manual word level gestures for entering text
on a soft-keyboard has been previously introduced in commercial
products such as Keymonk [10] and research efforts such as Bi et
al.’s bimanual gesture keyboard [2]. Bi et al. [2] showed that a
bi-manual gesture- based method can reduce the length of the stroke
used to draw a word into two shorter strokes which are drawn by the
thumbs. Our work builds upon these systems by allowing both thumbs
to draw small strokes over the keys on their side of the keyboard
without waiting their turn in the letter sequence of a word. Though
Bi et al. mention the possibility of concurrent input [p.144, 2],
this was not explored; the user touches the keys in a word
sequentially with both Bi et al.’s bimanual keyboard and
Keymonk.
Below, we discuss how our method builds upon other works. We review
prior research in gesture-based text input, thumb-based text input,
and two-handed text input methods for mobile devices.
2.1 Gesture-Based Text Input Virtual keyboards can enable
text-input on mobile devices that do not have a physical
mini-QWERTY keyboard. Methods that support text-entry using
gestures—either through touch or stylus— enable users to enter text
either at the character-level or at the word- level. A detailed
review of such techniques can be found elsewhere [19].
Character-level input techniques allow users to type a word by
making multiple stroke gestures that correspond to different
letters, one a time. The most common character-level technique is
Graffiti, created by Palm, Inc. The Graffiti alphabet resembles the
Roman letters, which makes it easy for users to recognize and learn
[4]. Likewise, unistroke maps a single stroke to a single character
[6]. Unistroke gestures do not resemble Roman letters, but are
designed to be well distinguishable from one another. One key
distinction of Graffiti and unistroke from other character-level
input techniques is that the user does not need to perform target
selection as a part of their text-entry. Castellucci and
MacKenzie’s evaluation [4] comparing Graffiti and unistroke shows
that participants were able to input 15.8 wpm using unistroke and
11.4 wpm using Graffiti; however, there were no statistical
differences in input speed between the two. EdgeWrite uses a
character set that resembles the Roman letters, but only requires
the user to touch corners of a box in a particular sequence as they
draw these gestures [33]. This enables for users with motor
problems to type using a lower keystroke per character (KSPC) than
with Graffiti. The Vector keyboard [11] and MessagEase [21] combine
unistroke gestures with target selection to simplify the gesture
set. Although no numbers were given within the paper [11], it can
be derived from the presented graph that 9 participants were able
to input text with the Vector keyboard at ~10.3 wpm. Using a Fitt’s
Law analysis of the MessagEase design, Nesbat [21] concludes that a
soft-key version of the keyboard potentially can support expert
text input performance between 30 and 37 wpm.
Word-level gesture-based input techniques, such as Cirrin [20] and
Quikwriting [23], allow users to enter a word with a single stroke.
Mankoff and Abowd [20] reported that one user was able to
achieve
20 wpm with Cirrin after using the technique for over 2 months;
Isokoski and Raisamo [9] showed that participants were able to
reach 16 wpm with Quikwriting using a stylus after 20 15-minute
sessions. In 2003, Zhai and Kristensson began to explore shape
writing as a way to allow users to perform shorthand gestures to
input words on an optimized keyboard layout [12][13][34][35]. This
concept has since been adopted by several commercial products for
small mobile touchscreen devices, such as ShapeWriter [36], Swype
[28], and SlideIT [26]. These virtual keyboards have been used on a
variety of platforms, including small devices, tabletops and
tablets. Expert performance of shape writing using a QWERTY layout
on tabletops have been computed to reach 40.7 wpm [24]; but have
not been computed for the tablet form factor. In Bi et al.’s study
[2], participants reached an average of 30 wpm using the unimanual
shape writing technique on a tablet. However, tablet users often
describe concerns of fatigue in the hand holding the device as well
as in the finger performing the gestures back and forth across the
large touchscreens [5][15].
2.2 Thumb-Based Text Input Although very few techniques have been
altered or designed specifically to support thumb-based text-input,
users often are able to effectively employ their thumbs to perform
input. For example, Silfverberg et al.’s study shows that
two-handed index finger and one-handed thumb use of multi-tap are
~27 wpm and ~25 wpm respectively [25]; their study also shows that
two-handed index finger and one-handed thumb of T9 is ~46 wpm and
41 wpm. One notable exception is Gonzalez et al.’s work [7] which
explored different ways of supporting text-entry while the user is
driving and her hands are on a steering wheel. Their work showed
that EdgeWrite [33] can be adapted to work on a Synaptics Stampad
that is embedded on steering wheel. The focus of these previous
works has been to examine how use of the thumb to support one-
handed interaction affects performance. Results from these works
show that performance is minimally affected, but provides other
benefits (such as it frees up the other hand, and it requires less
visual attention because the targets are in fixed location relative
to the thumb’s anchor position). An important implication when
designing thumb-based interactions, however, is to make target
sizes large enough that they can be accurately selected by the
thumb [22].
2.3 Two-Handed Text Input Because typing on a desktop QWERTY
keyboard is typically performed as a two-handed task, it is not
surprising that users can effectively employ both hands with many
existing text-entry methods. For example, MacKenzie and Soukoreff
[17] studied two handed (specifically thumb) use of physical mini
QWERTY keyboards and predicted that expert typing speed can reach
~61 wpm. Novel techniques for chording on a standard 12-key keypad
also have been developed to leverage two-handed input, such as
ChordTap [30].
On large touchscreen devices, the 1Line Keyboard is a technique
that supports touch-typing on a tablet with a reduced layout [16].
Most other work, such as LucidTouch [31], SwiftKey [27], Thumb
Keyboard [29], and Samsung’s S1 tablet support typing by dividing
the keyboard interface into two halves, allowing users to press the
keys with the hand closest to each. The 2-Thumb Gesture keyboard
presented in this paper, along with the previously mentioned
Keymonk [10] and Bi et al.’s bimanual gesture keyboard [2], differ
from these other works in that users employ their thumbs to draw
stroke gestures across the keys on each side of the keyboard
instead of performing individual key-presses. Bi et al. [2]
investigated two modes for completing the entry of a word:
finger-release and space-
required. They showed that after 1.5 hours, participants were able
to enter text at 26 wpm and 21 wpm with the finger-release and
space-required methods respectively. Although participants were
slower with the bimanual techniques, they perceived that it was
more comfortable and less physically demanding than a unimanual
gesture-based technique. The benefits and challenges associated
with two-handed input have been well studied [1][3][8][14], the
research question that this paper seeks to answer specifically is
how to extend previous work to enable the user to perform
simultaneous word drawing gestures on large touchscreen tablets
while holding the device in both hands.
3. DESIGN & IMPLEMENTATION The 2-Thumb Gesture soft keyboard is
a non-sequential bi-manual gesture based text input technique for
touch tablets. We developed 2TG as an input method editor (IME) for
Android OS version 2.3 or higher. In contrast to prior work
designed to optimize input with the tablet placed on a table (e.g.,
the 1Line keyboard [16], we designed 2TG for use while holding the
tablet (i.e., with the device cupped in both hands and using the
thumbs for input). In particular, we are hoping to enhance
experiences in which users must stand and hold the tablet (e.g., a
nurse taking notes on her rounds, a coach taking notes on the
field, a user typing on a bus). Similar to Bi et al.’s bimanual
keyboard [2] and Keymonk [10], the user draws two separate strokes
that together touch all the letters in a word. Because Bi et al.
have previously showed that the finger-release and space-required
gesture completion modes resulted in comparable entry speed, with
finger release being slightly faster,
we only explore finger-release in this work. Furthermore, we
explore allowing the user to draw the strokes non-sequentially,
that is, the thumbs do not need to wait until their turn in the
word’s letter sequence to continue drawing their respective stroke.
By allowing the two thumbs to non-sequentially draw short stroke
gestures on their side of the screen to touch all the letters in a
word, it is possible to also reduce the amount of time required to
enter a word.
3.1 Key Layout & Size Because the size and weight of a tablet
make it hard to hold single- handed for long tasks, we devoted
special attention to enabling the user to hold it steady with both
hands during text input. Given that both hands would be holding the
device, we designed the keyboard’s layout and size specifically to
support comfortable thumb-based input.
The 2-Thumb Gesture keyboard preserves the QWERTY layout to avoid
the overhead associated with learning a new letter arrangement. To
prevent the user from straining to reach any key with the
thumbs—which happens most often with keys towards the middle of the
keyboard (T, G, B, Y, H, and N), we enlarged these center keys,
similar to the Microsoft Natural keyboard. Thus, the layout of the
keys on both halves remains visually similar to traditional desktop
keyboards and allows easier access of these frequently used keys
(see Figure 2).
3.2 Input Language Users input text by drawing stroke gestures over
the letters sequentially (see Figure 3). However, because the
thumbs only
Figure 2. The 2-Thumb Gesture keyboard interface. The design draws
inspiration from the Microsoft Natural keyboard; it enlarges the
keys at the center (T, G, B, Y, H, N) to split the layout into two
distinct halves while preserving the familiar QWERTY layout. Keys
on each side are intended to be swiped only by the thumb on that
side of the screen.
Figure 3. Typing love as a novice user. The user starts by placing
her right thumb on the L key and then swipes it to O. Next, she
places her left thumb on V and then swipes it to E. Once she lifts
both thumbs from the keyboard, the keyboard interprets the gesture
into a list of candidate words and automatically selects the
highest frequency word.
draw on their side of the screen, the stroke gestures are
inherently different from the unistroke gestures supported by
counterpart systems. First, although a novice user may still swipe
her thumbs to different keys in the sequential order defined by the
respective letter’s appearance in the word, the strokes themselves
no longer visually capture the full sequential order of the keys
across both sides of the keyboard. This property of the gesture
language results in a feature that allows expert users who are
familiar with the gestures to input the strokes on each side of the
keyboard non- sequentially and even simultaneously (i.e., without
needing to wait for that side’s turn in the word’s letter
sequence). Figure 4 illustrates how an expert user would type love
using both thumbs.
3.3 Gesture Recognition Our method performs a dictionary lookup
based on the gestures drawn using each hand. However, because keys
for letters not found in a word are crossed while the user swipes
between the letters in the word, gestures must be interpreted as a
set of likely candidate words. As mentioned above, we removed all
entries in COCA that contained non-alphabetic characters as well as
those entries not found on dictionary.com. We then extracted the
list of keys that must be entered sequentially on the left side of
the keyboard as well as its right side equivalents. We next built
an SQLite database holding the word, its frequency, and the left
and right key sequences for entering it.
To determine the word the user entered through the thumb gestures,
the keyboard computes the list of key points from each stroke,
including the first and last points. Key points are points in a
gesture where the thumb changes its direction either vertically
(between up and down) or horizontally (between left and right). We
used these strict criteria to lower the false detection of key
points. However, this approach then only extracts from the strokes
a set of key sequences that partially matches those in the database
for the word that the user wants to enter. The input key sequence
entered may also partially match with the key sequences for a few
other words. The system retrieves a full list of the possible
candidates, sorts the responses based on the frequency of the words
in the COCA corpus, and displays the top four as “candidate words”
for the gestures, with the top one automatically selected and shown
in the editor field.
To evaluate the coverage of the words in COCA supported by our
algorithm, we computed the percentage of words that appear as the
most likely candidate for the associated gesture sequences, as well
as the percentage of words that appear as the 2nd, 3rd, and 4th
most likely candidates for those sequences. Again, these
calculations assume a perfect gesture that travels through the
center of each key in the word’s letter sequence. The results
showed that in the top
10,000 words, 97.19% can be typed without the need for
disambiguation and 99.96% appear in the top 4 most likely words for
any gesture sequences. Overall, 99.51% of the words in the entire
corpus appeared in the top 4 most likely words for its gesture
sequences (see Figure 5).
We note that other approaches for performing gesture recognition
can be adopted, such as an adaptation of the shape writing
algorithm [13] or machine learning. In this paper, we investigate
the user’s ability to learn and perform the strokes, without
introducing system support for inferring the user’s intent; this
allows us to then analyze how much effect adopting a different
gesture recognition method and relaxing the requirement for the
users to accurately and precisely draw a stroke could potentially
have on the user’s text input performance.
4. EVALUATION METHOD We evaluated the 2-Thumb Gesture technique in
a laboratory study focused on the learnability and usability of the
technique as a text input method on tablet computers, with Swype
[28] as a reference technique. As previously noted, in Bi et al.’s
study [2], participants reached an average of 30 wpm using the
unimanual shape writing technique on a tablet, while reaching
between 21 and 26 wpm using the bimanual approach. Similarly, we
included Swype as a reference technique to develop a rich
understanding of the advantages and drawbacks of both one-finger
and two-thumb shape writing on touch tablets.
We note that our decision to compare against a fully implemented
commercial unimanual shape writing technique has advantages as well
as drawbacks. We, of course, did not expect to outperform a fully
implemented Swype; however, we hoped to demonstrate that our
implementation’s performance is promising in relation to a fully
implemented system that has error-handlings. A drawback to this
decision is that it does not allow us to fully compare the 2-
handed approach in which keys do not have to be touch sequentially
against a 1 finger approach implemented using the same gesture
recognition method. That issue and controlling for other design
parameters (e.g., support for word prediction, typo detection/auto-
correction, and so forth) are outside of the scope of this specific
investigation and can be points for future studies.
4.1 Study Design & Procedure A 3-factor mixed design was used
with input technique (1-finger Swype or 2TG) and session (1, 2, or
3) as within-subjects factors, and prior experience (No Experience,
Prior Swype Experience) as
Figure 5. Coverage of the gesture recognition algorithm. Each line
shows the coverage of the words returned within the first, second,
third and fourth most likely candidate for its gesture
sequences.
Figure 4. Typing love as an expert user. The user starts by placing
her right thumb on L and her left thumb on V, and then
simultaneously drags her right thumb to O and her left thumb
V.
a between subjects factor. Presentation order was fully
counterbalanced and participants were randomly assigned to a
condition.
During the experiment, we asked participants to enter short phrases
presented on the screen as fast and accurately as possible using
the given text entry method. We prepared phrases based on MacKenzie
and Soukoreff‘s set [18]. We randomized the order of the phrases
and grouped them into six blocks of phrases. No phrase repeated
within a block.
Each person participated in three sessions. Consecutive sessions
were scheduled 2–72 hours apart around the participants’ schedules.
At the first meeting, participants also learned about and practiced
each text entry method.
Each session consisted of two 20-minute half-sessions (one 20-
minute half-session for each technique). Each half-session started
with three practice phrases, followed by the phrases pre-arranged
for that block. Participants had 20 minutes to type as many phrases
as they could. We counter-balanced the presentation order of the
text entry methods across the participants for the first session
and alternated their presentation for subsequent sessions with the
same participant.
4.2 Participants We used word of mouth, flyers, and online posts to
recruit right- handed participants. Because of procedural
inconsistencies, data for 2 participants were discarded from
analysis, leaving 10 participants (6 female and 4 male; with a
median age of 29 years, SD=5.3) for this experiment. All
participants had at least a high- school level of English literacy.
All participants reported no motor disability in their arms and
hands, and reported no visual disability. Half of the participants
reported having prior knowledge of how to use Swype (and either
currently used it or had used it in the past); the remaining 5
participants had no prior experience with the technique.
4.3 Apparatus We used an Asus Eee Pad Transformer for the text
entry interfaces. We asked participants to hold the tablet during
the study. For the Swype condition, we used the Beta 3.26 version
of the Swype soft keyboard. The height and width of the keys in the
Swype soft keyboard are the same as those in the 2-Thumb Gesture
keyboard (with the exception of t, a, g, h, and n keys, which are
wider for 2TG as shown in Figure 2).
5. RESULTS To analyze collected data, we modified Wobbrock and
Myer’s StreamAnalyzer software [32] to account for word-level
operations like word deletion and word corrections. We removed any
data point over 3SD from the average typing time and the average
error rate for each participant in each session as an outlier. In
total, 3.0% of the data points were removed.
We ran a 232 (technique session expertise) repeated measures ANOVA
on words-per-minute, and the corrected and uncorrected error rates.
In the presentation below, where df is not an integer, we have
applied a Greenhouse-Geisser adjustment for non-spherical data. All
pairwise comparisons were protected against Type I error using a
Bonferroni adjustment. Along with statistical significance, we
report partial eta-squared (η2), a measure of effect size. A
preliminary analysis, which included presentation order as a
between-subjects factor, yielded no significant main or interaction
effects involving order, giving us confidence that counterbalancing
the techniques sufficiently accounted for any learning or fatigue
effects.
5.1 Performance Overall, text input rate was impacted by input
technique, indicating that overall the more familiar 1-finger Swype
method outperformed the novel 2-Thumb Gesture technique (F1,8 = 47,
p = .0001, 2
= .855). At the 3rd session, participants averaged 32.4 wpm (SD:
8.51) with Swype and 24.43 wpm (SD: 4.29) with 2TG. However, there
was also a significant interaction between technique and expertise
(F1,8 = 16.29, p = .004, 2
= .671) Pair-wise comparisons revealed that the effect of technique
only held for those with prior Swype experience (p < .001). For
those with no prior experience with the techniques, no significant
difference was found (p = .081). There was, however, no main effect
of expertise (p =.079). Figure 6 shows words-per-minute for each
technique by level of expertise.
A main effect of session on words-per-minute (F1.22,9.72 = 44.02, p
< .0001, 2
= .846), confirmed by pair-wise comparisons (Session 1–2, p = .001;
Session 2–3, p < .001), showed that performance improved with
each session. This effect was consistent across techniques and
expertise, as shown in Figure 7 and indicated by the absence of
interaction effects (session technique: p = .271, session
expertise: p = .716, and session technique expertise: p =
.62).
The average uncorrected error rate (the rate of errors remaining in
the transcribed text, including insertion, substitution, and
deletion errors [32]) over all sessions was 0.4% (SD=1.8%) and 0.7%
(SD=3.2%) for the Swype and 2-Thumb Gesture techniques,
respectively. These low rates are not surprising once we look at
their corrected equivalents. On average, participants had a
corrected error rate of 9.8% (SD: 15.4%) with the Swype technique
and 14.8% (SD: 14.7%) with the 2-Thumb Gesture technique,
indicating few errors remained in the input, because participants
were correcting their errors in this study.
As with input rate, expertise and technique impacted the corrected
error rate, yielding a main effect of technique (F1,8 = 21.36, p =
.002, 2
= .727) and an interaction effect between technique and expertise
(F1,8 = 12.98, p = .007, 2
= .619). Pairwise comparisons revealed that those with prior Swype
experience had higher corrected error rates with 2TG (p < .001),
suggesting a possible negative transfer effect. There was no
difference between the techniques for those without Swype
experience (p = .492), as shown in Figure 8. There was also a
significant interaction between session and technique (F1,8 = 5.69,
p = .014, 2
= .416). Pairwise comparisons revealed that the main effect of
technique only held in sessions 1 (p < .001) and 3 (p = .001),
but not in session 2 (p = .11). In session 2, the error rate spiked
for the Swype technique, but remained consistent in 2TG. For
uncorrected error rate, no significant differences were found,
which is not surprising given the low rates observed.
5.2 Non-Sequential Bi-Manual Input We designed the 2-Thumb Gesture
technique so that both thumbs could non-sequentially draw stroke
gestures. The challenge with being able to perform text input with
the 2-Thumb Gesture keyboard in this manner is being able to
identify or recall the correct left and right gesture sequences for
a word, behavior that we expect to develop with practice.
As expected, participants initially found the idea of performing 2-
Thumb Gesture to be daunting.
“It seemed hard at first to kinda cut the words and type them in
halves. I had to be more conscious of what each thumb was supposed
to do. But it’s a cool idea and technique. I got used to it.” –P12
(22, F)
Thus, when entering a word for the first time, the thumbs would
each draw a stroke over the keys on their side of the keyboard to
sequentially spell each letter in a word. However, an interesting
outcome of the initial mental and motor requirements associated
with learning the 2-Thumb Gesture method is how participants began
to remember the gestures, more so than with Swype. Participants
commented that they began to remember gestures for short words such
as “the” and “this.”
“I liked the two hand method better for shorter words! I know the
gestures for words like ‘the’ and ‘water.’” –P2 (23, F)
Not surprisingly, as participants began to learn the gestures for
short words and common letter sequences, they also began to draw
both strokes simultaneously, touching keys out of order with
respect to their turn in the actual letter sequence. Participants
perceived that doing so improved their input rate.
“I don’t have to do things simultaneously, but I feel like I would
be slower [sequentially].” –P2 (23, F)
At the 3rd session, 19.0% (SD: 8.0%) of the input happened in this
manner. We expect that as participants begin to remember more
gestures over time, they will continue to improve their text-input
rates. For example, participants commented that for longer words,
they had begun to remember the gestures for portions that made up
of common letter sequences, such as “ing” and “pre.”
5.3 User Feedback Although mentally dividing a word into a two
stroke gesture seemed difficult to the participants initially,
participants started to describe it as being “intuitive” and even
fun to use as they continued to learn the technique,. For example,
after initially struggling to use the 2-Thumb Gesture keyboard for
the first 5 minutes, P7 (38, M) declared “I got it.” As a person
without prior knowledge of Swype, P7 was able to learn both at the
same rate; however, he preferred the 2-Thumb approach.
“Dude, this (Swype) sucks in comparison to the new school way
(2TG).”—P7 (38, M)
“The two hand method seems more intuitive. Even though two hand is
less accurate, the user experience feels enhanced.” – P5 (30,
M)
Furthermore, although a few participants suggested ways of
improving the comfort of 2TG (e.g., by changing the placement and
size for some of the keys), participants uniformly reported
discomfort when using Swype. In particular, participants noted that
holding the tablet with one hand can be tiring on the hand and the
wrist.
“Oh my g-d, my hand!” –P2 (23, F)
“This gets really heavy after a while! My hand…the one that holds
the tablet is dead. The fatigue is definitely less with the two
hand method.” –P4 (31, F)
“After 5 minutes or so, my finger started hurting and my wrists
were cramping!” –P11 (20, F)
However, all participants described the Swype technique as being
“more forgiving” and some indicated that with time, the two handed
technique might improve.
“Two hand seems more intuitive, but the technology is not quite
there.” –P5 (30, M)
Figure 6. Words per minute for 2 Thumb Gesture and Swype, by prior
experience with Swype. Error bars show 95% confidence intervals
(N=10).
Figure 7. Words per minute across each session, by technique and
experience with Swype. Error bars show 95% confidence intervals
(N=10).
Figure 8. Corrected error rate for 2 Thumb Gesture and Swype, by
prior experience with Swype. Error bars show 95% confidence
intervals (N=10).
Participants noted that the commercial Swype keyboard always tried
to guess what they were gesturing. For example, the participants
could miss some keys (instead only touching their neighboring keys)
when performing a gesture and the system may still return what was
intended as a potential candidate word. Furthermore, they described
being able to continue with a gesture and correct it even if they
messed up. Although the gesture recognition approach that we
adopted also allows the participants to adjust their gesture if it
is incorrect, it requires the participants to accurately touch all
the keys in the word.
By developing and evaluating a keyboard without any tolerance for
input error, we were able to gain an understanding of the
effectiveness of the 2-Thumb Gesture technique by itself alone.
Thus, the results of the study demonstrate the feasibility of the
method. Though 2TG did not outperform Swype, adding tolerance for
input error to our method (similar to that already provided by the
commercial Swype technique used as a comparison) can only improve
its user performance.
6. DISCUSSION In this section, we discuss some of the factors that
could have affected the text input rate observed in the study.
Additionally, we discuss some possible ways to improve upon this
work.
6.1 Input Errors Participants made two types of errors when using
2TG that limited their input rates. First, the system rejected some
of the participants’ gestures. Whenever the system was unable to
recognize a gesture, the participant had to perform that gesture
again. Second, much like typing on a normal keyboard, participants
sometimes entered typos, which resulted in the keyboard returning
the wrong words. The design of our recognition algorithm did not
recognize and fix typos automatically. The low uncorrected error
rate observed indicates that participants corrected these typos,
thereby reducing the input rate.
6.1.1 Rejection Errors The system rejected 5.9% (SD: 2.6%) of the
gestures inputted by participants. Although 5.9% is a low rate, it
has a noticeable effect on the participants.
“I re-enter text more in 2TG because when it's wrong, there aren't
any choices.” –P3 (31, F)
In contrast, participants did not report experiencing any rejection
errors when using the Swype technique. All participants commented
that they wished the technique would be as forgiving as Swype,
which did not require them to directly touch all the keys but only
be in proximity of those keys. Future versions of the keyboard can
relax the requirement that the users must accurately touch the keys
in the gesture sequence and apply a proximity threshold against
which it accepts user input.
A common problem that many participants encountered was touching
keys towards the middle of the keyboard (T, G, B, Y, H, and N) with
the opposite hand. Our implementation only allows users to touch
keys from one side of the keyboard with the respective thumb.
However, participants mentioned that they typically do not abide by
such a strict input model when using desktop keyboards. As a
result, while entering text with the 2- Thumb Gesture keyboard,
there was a tendency for them to reach for some keys with the
opposite thumb.
“I have to think of the halves of the keyboard. My biggest problem
was that I had to divide the keyboard in my head and I couldn’t go
over.” –P2 (23, F)
Because our system disallowed this behavior, participants’ input
gestures were rejected by the recognizer when they included keys
from the other half of the screen. To address this problem, the
gesture database can be built to treat the middle keys as ones that
can be input as a part of the gestures from either side of the
screen. Alternately, the keyboard could be further split, leaving a
space with invisible keys in between, as on the iOS5 split keyboard
and Bi et al.’s bimanual keyboard [2].
6.1.2 Typos Overall, the average corrected error rate for 2TG was
14.9%. In this work, the corrected error rate reflects the rate at
which two types of actions could have occurred: 1) the manual
deletion of incorrectly typed text and re-entering correct text,
and 2) the automatic correction of incorrect text that was returned
by the gesture recognizer through the selection of another word
from the candidates list (i.e., the user performed a disambiguation
action). As mentioned earlier, for the 10,000 highest frequency
words in the COCA, 97.19% can be typed with the 2-Thumb Gesture
keyboard without the need for disambiguation and 99.96% appear in
the top 4 most likely word for any gesture sequences. When we
examine the disambiguation rate, at the 3rd session, participants
on average performed disambiguations for only 2.7% (SD: 1.4%) of
the text that they entered. This means that the remaining amount of
the corrected errors (~12.2%) resulted from correcting typos.
By developing and evaluating a keyboard without any tolerance for
input error, we were able to gain an understanding of the
effectiveness of the 2-Thumb Gesture technique by itself alone. The
results from our study establish that the 2-Thumb Gesture keyboard
is a feasible method that can be learned and used. Adding tolerance
for input error to the 2-Thumb Gesture keyboard can only improve
its user performance. With feasibility established, an obvious next
step is to collect the data necessary to model common input errors.
For example, participants sometimes touch keys from the same side
of the keyboard in the wrong order (e.g., gesturing EV instead of
VE for LOVE), or miss the last key by a few pixels (undershooting
the target). Using such data, we can develop an error model that
can be used to automatically correct typos.
6.2 Learnability Participants commented that the visual feedback
provided by our keyboard implementation currently does not help
them remember the gesture patterns for long words.
“I think one of the problems is that for long words, you start
losing track of what (the letters in) the words are.” –P5 (30,
M)
Currently, our keyboard implementation paints the full gestures
until both thumbs are lifted. In future versions of the software,
we will design the keyboard to better highlight parts of the
gestures in which high-frequency letter sequences were inputted. As
participants learn the common gestures, we can begin to adapt the
visualization to continue to teach them the lower frequency letter
sequences as well.
7. CONCLUSION In this paper, we presented the design and evaluation
2-Thumb Gesture, a technique that breaks the input of a word into
two smaller stroke gestures that can be performed non-sequentially
by both thumbs on each side of the keyboard. Our evaluation results
show that after 40-60 minutes of use, participants were able to use
the 2- Thumb keyboard to enter text at 24.43 wpm, with an
uncorrected error rate of 0.65% and a corrected error rate of
14.9%. These
overall average performance results were similar to those reported
by Bi et al. [2], who also showed that participants reported a
difference in comfort and physical demand between the unimanual and
bimanual input approaches; similarly, our study confirmed
participants were able to hold and use the tablet with both hands
without the substantial fatigue that results from the one-handed
approach.
Beyond those results, our implementation and study show that at the
third session, participants had begun to learn the gestures for
many short words and were able to perform on average 19.0% (SD:
8.0%) of the input by simultaneously gesturing with both thumbs and
touching keys out of order with respect to their turn in the actual
letter sequence while only needing to disambiguate 2.7% of their
input. As participants begin to remember more gestures over time,
they will continue to improve their text-input rates.
Additionally, our study shows that the learning and use of the 2-
Thumb Gesture technique to perform text input was comparable to
that of the commercial Swype technique by those who had no prior
experience with either method. Although input by those with prior
knowledge of Swype was faster using Swype than 2TG, their learning
and performance rates with our two-thumb version were not
statistically different from those with no prior Swype knowledge.
Furthermore, the current 2-Thumb Gesture keyboard does not support
erroneous input in the same manner as the commercial Swype
keyboard, against which it was evaluated. It is important to note
that these results were achieved despite the fact that at the 3rd
session, 18.1% of the participant’s input were errors, which
required the user to re-input the text (5.9% were gestures rejected
by the recognizer; 12.2% were typos not recognized by the
system).
For future work, we will improve the interface to address issues
that prevented the participants’ input rates from being closer to
the predicted value. First, we aim to help users learn how to input
many common letter sequences by modifying the way that the strokes
are drawn on the screen to highlight those gestures. Additionally,
we will modify the interface to give users more freedom for how to
input letters towards the middle of the keyboard and to remove the
requirement that users must directly touch the keys (to make the
technique more “forgiving” like Swype). Finally, we will develop a
model capturing how users incorrectly type different words in order
to automatically correct typos.
8. REFERENCES [1] Balakhrishnan, R., Hinckley, K. Symmetric
bimanual interaction. In
Proc. of CHI 2000, ACM Press, 33-40. [2] Bi, X., Chelba, C.,
Ouyang, T., Partridge, K., Zhai, S. Bimanual
Gesture Keyboard. In Proc. of UIST 2012, ACM Press, 137-146. [3]
Buxton, W., Myers, B. A study in two-handed input. In Proc. of
CHI
1986, ACM Press, 321-326. [4] Castellucci, S.J., MacKenzie, I.S.
Graffiti vs. unistrokes: an empirical
comparison. In Proc. of CHI 2008, ACM Press, 305-308. [5]
Enterprise Efficiency. Swype Makes Tablets Interesting.
http://www.enterpriseefficiency.com/messages.asp?piddl_msgthread
id=231470
[6] Goldberg, D., and Richardson, C., Touch-typing with a stylus.
In Proc. of INTERCHI 1993, ACM Press, 80-87.
[7] González, I.E., Wobbrock, J.O., Chau, D.H., Faulring, A.,
Myers, B.A. Eyes on the road, hands on the wheel: thumb-based
interaction techniques for input on steering wheels. In Proc. of
Graphics Interface 2007, 95-102.
[8] Guiard, Y. Asymmetric division of labor in human skilled
bimanual action: the kinematic chain as a model. Journal of Motor
Behaviour, 19(4), 1987, 486-517.
[9] Isokoski, P., Raisamo, R. Quikwriting as a multi-device text
entry method. In Proc. of NordiCHI 2004, ACM Press, 105-108.
[10] Keymonk. http://keymonk.com/ [11] Klima, M., Slovacek, V.:
Vector Keyboard for Touch Screen Devices.
In Proc. of HCII 2009: Ergonomics and Health Aspects of Work with
Computers, 250–256.
[12] Kristensson, P.O. Discrete and Continuous Shape Writing for
Text Entry and Control. Ph.D. Dissertation, Department of Computer
and Information Science, Linköping University (Linköping, Sweden),
2007.
[13] Kristensson, P.O., Zhai, S. SHARK2: a large vocabulary
shorthand writing system for pen-based computers. In Proc. of UIST
2004, ACM Press, 43-52.
[14] Leganchuk, A., Zhai, S., Buxton, W., Manual and Cognitive
Benefits of Two-Handed Input. TOCHI, 5(4), 1998, 326-359.
[15] Le Pan Life. Multi-touch Screen.
http://lepanlife.com/forum/showthread.php?tid=266
[16] Li, F.C.Y., Guy, R.T., Yatani, K., Truong, K.N. The 1line
keyboard: a QWERTY layout in a single line. In Proc. of UIST 2011,
ACM Press, 461-470.
[17] MacKenzie, I.S., Soukoreff, R.W. A model of two-thumb text
entry. In Proc. of Graphics Interface 2002, 117-124.
[18] MacKenzie, I.S., Soukoreff, R.W. Phrase sets for evaluating
text entry techniques. In Ext. Abs. of CHI 2003, ACM Press,
754-755.
[19] MacKenzie, I.S., Soukoreff, R.W. Text entry for mobile
computing: Models and methods, theory and practice. Human-Computer
Interaction, 17, 147-198.
[20] Mankoff, J.C., Abowd, G.D. Cirrin: A Word-Level Unistroke
Keyboard for Pen Input. In Proc. of UIST 1998, ACM Press,
213-214.
[21] Nesbat, S.B. A system for fast, full-text entry for small
electronic devices. In Proc. of ICMI 2003, ACM Press, 4-11.
[22] Parhi, P., Karlson, A.K., and Bederson, B.B. Target size study
for one- handed thumb use on small touchscreen devices. In Proc. of
MobileHCI 2006, ACM Press, 203-210.
[23] Perlin, K. Quikwriting: Continuous Stylus-Based Text Entry. In
Proc. of UIST 1998, ACM Press, 215-216.
[24] Rick, J. Performance optimizations of virtual keyboards for
stroke- based text entry on a touch-based tabletop. In Proc. of
UIST 2010, ACM Press, 77–86.
[25] Silfverberg, M., MacKenzie, I.S., Korhonen, P. Predicting text
entry speeds on mobile phones. In Proc. of CHI 2000, ACM Press,
9-16.
[26] SlideIT. http://www.mobiletextinput.com/ [27] SwiftKey.
http://www.swiftkey.net/ [28] Swype. http://www.swype.com/ [29]
Thumb Keyboard.
http://www.beansoftapps.com/archives/products/thumb-keyboard [30]
Wigdor, D., Balakrishnan, R. A comparison of consecutive and
concurrent input text entry techniques for mobile phones. In Proc.
of CHI 2004, ACM Press, 81-88.
[31] Wigdor, D., Forlines, C., Baudisch, P. Barnwell, J., Shen, C.
Lucid touch: a see-through mobile device. In Proc. of UIST 2007,
ACM Press, 269-278.
[32] Wobbrock, J.O., Myers, B.A. Analyzing the input stream for
character- level errors in unconstrained text entry evaluations.
TOCHI, 13(4), ACM Press, 2006, 458-489.
[33] Wobbrock, J.O., Myers, B.A., Kembel, J.A. EdgeWrite: a stylus-
based text entry method designed for high accuracy and stability of
motion. In Proc. of UIST 2003, ACM Press, 61-70.
[34] Zhai, S., Kristensson, P.O. Shorthand writing on stylus
keyboard. In Proc. of CHI 2003, ACM Press, 97-104.
[35] Zhai, S., Kristensson, P.O. The Word-Gesture Keyboard:
Reimagining Keyboard Interaction. Communications of the ACM, 55(9),
2012, 97-104.
[36] Zhai, S., Kristensson, P.O., Gong, P., Greiner, M., Pen, S.,
Liu, L., & Dunnigan, A. Shapewriter on the iPhone: from the
laboratory to the real world. In Ext. Abs. of CHI 2009, ACM Press,
2667-2670.
9. VIDEO