An evaluation of four patterns of interaction for integrating disparate ...
UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development
30
UsiGesture: an Environment for Integrating Pen-based Interaction in User Interfaces 17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 1 François Beuvens and Jean Vanderdonckt [email protected] Researcher at LiLab, http://www.lilab.be Université catholique de Louvain (Belgium)
-
Upload
jean-vanderdonckt -
Category
Technology
-
view
1.089 -
download
1
description
— Several algorithms have been developed for pen-based gesture recognition. Yet, their integration in streamline engineering of interactive systems is bound to several shortcomings: they are hard to compare to each other, determining which one is the most suitable in which situation is a research problem, their performance largely vary depending on contextual parameters that are hard to predict, their fine-tuning in a real interactive application is a challenge. In order to address these shortcomings, we developed UsiGesture, an engineering method and a software support platform that accommodates multiple algorithms for pen-based gesture recognition in order to integrate them in a straightforward way into interactive computing systems. The method is aimed at providing designers and developers with support for the following steps: defining a dataset that is appropriate for an interactive system (e.g., made of com¬mands, symbols, characters), determining the most suitable gesture recognition algorithm depending on contextual variables (e.g., user, platform, environment), fine-tuning the parameters of this algorithm by multi-criteria optimization (e.g., system speed and recognition rate vs. human distinguishability and perception), and incorporation of the fine-tuned algorithm in an integrated development environment for engineering interactive systems.
Transcript of UsiGesture: an Environment for Integrating Pen-based Interaction in User Interface Development
1
UsiGesture: an Environment for Integrating Pen-based Interaction in User
Interfaces
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
François Beuvens and Jean Vanderdonckt [email protected]
Researcher at LiLab, http://www.lilab.beUniversité catholique de Louvain (Belgium)
2
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
3
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
4
Context- Scope: 2D pen-based gestures- Needs of pen-based gesture recognition
- PDAs – Graffiti- Mouse gestures plug-in for Firefox- Games- …
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
5
Motivations- No « One fits all » algorithm- Needs of algorithms comparison
- Same data and conditions- Data acquisition
- Enhancements of existing algorithms
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
6
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
7
Gesture database acquisition
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
8
Gesture database acquisition- 30 participants
- 16 men and 14 women with average age of 28- 60 gestures to reproduce
- 26 lower case letters, 10 digits, 16 action commands, 8 geometrical shapes
- 10 reproductions of each gesture by participant- More information:
https://sites.google.com/site/comparativeevaluation/data-collection
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
9
Comparison study: analysis criteria- Training types
- User dependent, user independent (2 methods)
- Gesture types- Letters, digits, actions, geometrical shapes
- Algorithms- Rubine, One Dollar, Levenshtein, stochastic Levenshtein
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
10
Rubine- Training
- Compute mean vector of features and weights for learning classes
- Recognition- Compute vector of features- Scalar product of weights
and vector- Best score is chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
11
One Dollar- Training
- Preprocessing: resampling, rotation, rescaling, translation
- Recognition- Preprocessing- Comparison to all training examples
- Finding optimal angle- Evaluating point-to-point Euclidean distance- Smaller distance chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
12
Levenshtein- Training
- Transformation into strings
- Recognition- Comparison to all training examples
- Computing Levenshtein distance- Smaller distance chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
13
Stochastic Levenshtein- Training
- Transformation into strings- Creation of a string pairs set- Optimization of the primitive edition costs
- Expectation-maximization algorithm
- Recognition- Comparison to all training examples
- Evaluating distance based on probabilities- Smaller distance chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
14
Results – initial version- Training types for Levenshtein on digits
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
15
Results – initial version- Gesture types for Rubine with user dependent
training and user independent training
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
16
Results – initial version- Algorithms on digits and actions with user
independent training
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
17
Results – initial version- Summary
- More Information: https://sites.google.com/site/comparativeevaluation/results
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
18
Extensions- Rubine
- Preprocessings
- One Dollar- kNN
- Levenshtein- Preprocessings, kNN, normalisations, multi-strokes gestures,
other features handling, costs matrix
- Stochastic Levenshtein- Idem except costs matrix
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
19
Results - Rubine’s extensions- Preprocessings: resampling, rotation, rescaling
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
20
Results - One Dollar’s extensions- kNN
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
21
Results - Levenshtein’s extensions- Normalizations, pre-processings, kNN, multi-
strokes, pressure inclusion, costs
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
22
Results - Stochastic Levenshtein’s extensions- Normalizations, preprocessings, kNN, multi-
strokes, pressure inclusion
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
23
Results – extensions- Summary
- More Information: https://sites.google.com/site/comparativeevaluation/results
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
24
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
25
Woke example (before)
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
26
Woke example (after)
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
27
WindowBuilder integration (demo)
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
28
Roadmap- Context- An open platform for benchmarking gesture
recognition algorithms- Including the results in user interfaces- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
29
Ongoing and future works- Theoretical
- Conceptual modeling of gesture-based UIs
- Methodological- Integration into UsiXML- Definition of development lifecycle (e.g. SPEM)
- Empirical- User study on user satisfaction with gesture integration - Development benefits estimation
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
30
Thank you for your attention!
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science
![[MOBILEHCI2013] MagPen: Magnetically Driven Pen Interaction On and Around Conventional Smartphones](https://static.fdocuments.net/doc/165x107/55a2ba141a28ab3f5f8b45e5/mobilehci2013-magpen-magnetically-driven-pen-interaction-on-and-around-conventional-smartphones.jpg)
