Post on 22-Dec-2015
1
The subArctic Input System
and
Extensions for Handling Inputs with Ambiguity
2
subArctic A Java-based GUI toolkit that I
(along with Ian Smith) built and distributed in 1996-97
Goal: highly extensible allowing support for lots of cool new interaction techniques– Emphasis on making new and strange
widgets / components / interactors easy to create
– “High ceiling”
3
Parties involved with a toolkit
Toolkit designer (me)
Interactor designer
Interface programmer
User
4
Parties involved with a toolkit
Toolkit designer (me)
Interactor designer
Interface programmer
User
Most toolkits target support
here
5
Parties involved with a toolkit
Toolkit designer (me)
Interactor designer
Interface programmer
User
By moving work up (into reusable library)
6
Parties involved with a toolkit
Toolkit designer (me)
Interactor designer
Interface programmer
User
But typically don’t help much here
(assume a fixed library)
7
subArctic
Toolkit designer (me)
Interactor designer
Interface programmer
User
SA tries to move work for many kinds of interactors into toolkit infrastructure
8
subArctic
Toolkit designer (me)
Interactor designer
Interface programmer
User
SA tries to move work for many kinds of interactors into toolkit infrastructure
Input system is a big part of that
9
Schema for pretty much all GUIs
init();
for (;;) {evt = wait_for_next_event();
dispatch(evt);
if ( damage_exists() ) redraw();
}
10
Schema of a GUI
init();
for (;;) {evt = wait_for_next_event();
dispatch(evt);
if ( damage_exists() ) redraw();
}
Event Record – recording of the relevant facts about
some occurrence of interest (i.e., user has manipulated
an input device)
11
Schema of a GUI
init();
for (;;) {evt = wait_for_next_event();
dispatch(evt);
if ( damage_exists() ) redraw();
}
Send (“dispatch”) the event to the object(s) that want it and/or know
how to respond to it(e.g., widget/component/interactor)
12
Event dispatch
All the work happens here Typically delegated to interactors
– E.g., buttons know how to respond to press and release like buttons should
– Each object keeps track of its own state
... but which interactor gets it Toolkit “event dispatch” process
13
Event dispatch policies
Two primary ways to decide which interactor gets an event
What are they?
14
Event dispatch policies
Two primary ways to decide which interactor gets an event– Positional dispatch
Based on where mouse is pointingExamples…
– Focus-based dispatchDesignated object always gets inputExamples…
15
Pop quiz
Should input for dragging be dispatched via positional or focus?
16
Pop quiz
Should input for dragging be dispatched via positional or focus?
Answer: No! (both)
17
subArctic input policies
subArctic encapsulates these “ways of dispatching inputs” in “dispatch policy objects”– Manages bookkeeping (e.g., picking)– Extensible set
Turns out there are other useful policies (e.g., for modal dialogs)
18
When interactors get events…
… they typically respond to them with the equivalent of a simple finite state machine
Press
Move
Release
19
subArctic has lib of common FSMs
Move a lot of input handling work typically done by interactor programmer up into the toolkit
One (highly parameterized) FSM for all – Brad’s “interactor” model (awful terminology :-)
Many customized FSM (extensible set)– subArctic input model
20
FSMs moved to toolkit object
“Dispatch agent”
Translates low level input into higher level terms
21
Dispatch agent example: move_drag
Translated to calls in input protocol:– drag_start(); – drag_feedback(); – drag_end();
With useful parameters (e.g. new pos)
Press
Move
Release
22
Dispatch agent example: move_drag
Translated to calls in input protocol:– drag_start(); – drag_feedback(); – drag_end();
With useful parameters (e.g. new pos)
Press
Move
Release
Defined by Java interface
23
Set of dispatch agents is extensible E.g., can subclass for
specialized kinds of drag such as “drag_within_box” or “snap_drag”– Can create custom for one interface– Once created can reuse
24
How it all goes together
Focus Policy
Positional Policy
Etc…Events
Press
Click
Rollover
Etc...
Etc...Text
Move drag
Grow drag
Etc...
25
How does interactor indicate it wants / can handle some type of input? “… implements input_protocol”
– Where “input_protocol” is interface with calls like drag_start(), etc.
For positional that’s it! For focus-based must also ask
for focus
26
Example: Hypertext for all
User (Ken Anderson) wanted to add hyperlinks to all objects– Hold down the control key and click– His external hyperlink database
would take over and map interactor id to hyperlink target
– But… how do you change every interactor to do this?
27
Example: Hypertext for all
In Swing, Motif, etc. this is essentially impossible
In SA, just insert a new subclass of the “click” dispatch agent that checks for the control key down– About 15 lines of code– Works for interactors written later!
28
Questions about the SA input system?
29
Providing Toolkit Level Support for Handling Ambiguity in Recognition-Based Input
See: http://doi.acm.org/10.1145/354401.354407and http://doi.acm.org/10.1145/332040.332459
Jennifer Mankoff, Gregory Abowd
Georgia Institute of Technology
Scott HudsonCarnegie Mellon University
30
Motivation
Recognition-based input offers the promise of naturalistic input modalities, BUT…
31
Motivation
Recognition-based input offers the promise of naturalistic input modalities, BUT…
Recognizers are imperfect– affects users– breaks current system models
New interfaces & mechanisms
32
Example Interaction
SILK
Hand-sketched interactors
33
Example Interaction
SILK
Interface developer can replace interactors with best recognition result
A button
34
Example Interaction
Correction Dialog (mediator)
test
36
Example Interaction
Problems with dialog – Not reusable or customizable– Hard to grow your own
Basically we don’t have toolkit support for recognition based UI
37
Motivation (cont.)
At much the same stage we were at for GUIs in 1983– No common model for input– No re-use
Infrastructure“widget library”
38
An alternative: Burlap
VIDEO
39
Goals of This Work
Robust, reusable infrastructure Reusable library Integrate with convent. toolkit
– Don’t throw out the baby with the bathwater
40
Talk Roadmap
Requirements for handling uncertain input
Extending toolkits to handle it Interaction techniques for
ambiguity Implementation
41
Invoking Application Actions
Action often done by callbacks– Direct procedure call to application
Hierarchical events are alternate approach– Delivered to app as well as toolkit
42
Hierarchical Events
Low-level events contribute to production of higher-level events
[Green TOG ‘86; Myers & Kosbie CHI ‘96]
User Inputcircle
stroke
down drag up• • • • • •
Corresponding Events
43
Implicit Assumption of Certainty
Implicit in all this is the assumption that the events really happened as reported
Problems arise when this isn’t true– E.g., brittle dialogs
44
Needed to Handle Uncertainty:
Allow for (and explicitly model) multiple alternatives– alternative higher level events– in recognition context: interpretations
Detect conflicting interpretations Mediation of conflicts
45
Needed to Handle Uncertainty:
Lexical feedback about uncertain events – split “feedback” from “action”
Library of mediators
46
How do we do this...
47
Extended Event Model
Uncertainty results in multiple interpretations
interpretation graph
Uncertain Input
circlebox
stroke
down drag up• • • • • •
circle
stroke
down drag up• • • • • •
Certain Input
48
Toolkit Extensions
Toolkit’s job is still to deliver events to objects– Now delivered to recognizers,
interactors, and application objects
Button
Checkbox Menu
Recog
49
Toolkit Extensions
Toolkit’s job is still to deliver events to objects– Objects initially only produce
(reversible) feedback, no actions
Button
Checkbox Menu
Recog
50
Another Change:
Interface Appearance Uncertain Event Hierarchy
circlebox
stroke
down drag up• • • • • •
Events dispatched to all who might use it
51
Details: Arranging for Mediation
Identify any conflicts Look for a mediators
– Pluggable list of them in toolkit Mediator chosen by meta-
mediator Mediator can:
“Pass”, “Pause”, “Accept”
52
Doing Mediation
Example:User selects interpretation
circle
box
circle
53
Doing Mediation (cont.)
Mediator prunes interpretation graph to tree
– App informed of accept & reject
circlebox
stroke
down drag up• • • • • •
circle
stroke
down drag up• • • • • •
54
Mediation Strategies
Many mediation strategies– e.g., Automatic vs. user involvement
Toolkit is fully “pluggable” (!)– Library of mediators provided, but– Can extend/build new ones as needed
Research goal:Finding new ones
55
Providing a Library of Mediators
56
Providing a Library of mediators
Survey of existing techniques [Abowd & Mankoff GVU Tech Report 99]
Three major classes Explored additional techniques
57
Providing a Library of mediators
Survey of existing techniques Three major classes
– Repetition
58
Providing a Library of mediators
Survey of existing techniques Three major classes
– Repetition– Choice:
Ripe for toolkit support Presentation form Instantiation time Contextual information Interaction Feedback
59
Providing a Library of mediators
Survey of existing techniques Three major classes
– Repetition– Choice– Automatic
ThresholdsConfusion matrixPlug in machine learning?
60
Providing a Library of mediators
Survey of existing techniques Three major classes Explored additional techniques
61
Example: Target Ambiguity
Problem: There may be multiple targets of a user action
Example: clicking– Kabbash (95)– Worden (97)
62
Example: Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking Solution: Give the user
a choice of all of the targets
QuickTime™ and a decompressor
are needed to see this picture.
63
Example: Target Ambiguity
Problem: There may be multiple targets of a user action
Example: Clicking Solution: Give the user
a choice of all of the targets
Other applications:– Any interface
involving mouse press/release
– Requires separation of concerns
– Works with all interactors
64
Implementation
Added support for mediation; ambiguity to subArctic toolkit– Reusable– Fully “pluggable”– Full existing library still works as is (!)
Small library of mediators Also working on non-GUI toolkit (
http://dx.doi.org/10.1145/572003)
66
Conclusions
Reusable infrastructure to support ambiguous input– Reduces difficulty of creating UIs– Easier to explore new design space
Done by modifying a toolkit, not a separate mechanism– Integrated with conventional input – Other support from toolkit still useful
67