Lecture 14: Model-based tools: Creating the UI Automatically Brad Myers 05-830 Advanced User...
-
Upload
eunice-holmes -
Category
Documents
-
view
216 -
download
0
Transcript of Lecture 14: Model-based tools: Creating the UI Automatically Brad Myers 05-830 Advanced User...
Lecture 14:Model-based tools:
Creating the UI Automatically
Brad Myers
05-830Advanced User Interface Software
1© 2013 - Brad Myers
Schedule of Lectures
Everyone OK? See also instructions off the homeworks page Right away: Check to make sure there are 2-
3 articles for each person
© 2013 - Brad Myers 2
Model-Based Tools
Overview Programmer describes the operation of the
system or the user interface in a specification language = the "model". model is a high-level description usually declarative (listing parts and relationships)
System automatically creates the interface Uses a low-level toolkit for the widgets
3© 2013 - Brad Myers
Goals: High-level description of an interface is easier to write than low-level
toolkit code Automatic generation may produce better UIs than programmers Allow separation of UI design (embodied in rules) from UI contents
(supplied by the programmer) Support dynamic creation of objects
define templates or prototypes Increase re-use since design rules shared by multiple applications Tools can reason over the specification to produce extra stuff:
Automatic generation of help, undo, etc. Transform interface into different but functionally equivalent interface Enabling and disabling of widgets Enforcement or checking of design guidelines- consistency,
completeness Enforces consistency since rules will pick similar objects for similar situations
Automatic adjustment to different screen sizes, etc., since rules can take this into account
Automatic analysis for quality NGOMSL analysis : GLEAN (Kieras, UIST'95)
4© 2013 - Brad Myers
Overview, cont. Related to the "Declarative" approach discussed
in previous lecture but here system has some intelligence or knowledge
so less has to be specified by the programmer. Different types:
Dialog box creators: Mickey, DON, Jade (lots of others)
Representations of the full UI: ITS, UIDE, Humanoid, MasterMind,
Newer: (XML-based) ConcurTaskTrees, Xweb, ICrafter, Supple, PUC
5© 2013 - Brad Myers
Dialog Box Creators
Easiest part of the UI to create Given a list of the contents, automatically:
1. choose widgets: specify type of desired input:
string = text input fieldnumber = sliderone-of-many = radio buttons or pop-up optionsmany-of-many = check boxes or checks in a menucommands = menu
6© 2013 - Brad Myers
Dialog Box Creators, cont.
2. arrange widgets based on look-and-feel guidelines
where OK goes which commands go in which menus
based on good graphic design principles.
3. set variables to reduce the number of callbacks necessary
7© 2013 - Brad Myers
Example: Mickey Programmer specifies UI by
putting special comments in aPascal file.
Uses the Apple Macintoshguidelines
Pre-processor to parse thePascal code and generate theMacintosh resources.
Maps Procedures into Menuitems. If parameter is one of a standard
set, pops up appropriate dialogbox or waits for input File to be read, file to be written New point, line or rectangle
8
© 2013 - Brad Myers
Mickey, cont. Variables:
Enumerated types mapped tocheck lists separated by lines. Sets the variables when changed.
Enumerated types with 2 choices mapped to name changes
Booleans: single checked items
9© 2013 - Brad Myers
Jade Brad Vander Zanden and Brad A. Myers, "Automatic, Look-and-Feel
Independent Dialog Creation for Graphical User Interfaces," Proceedings SIGCHI'90: Human Factors in Computing Systems. Seattle, WA, April 1-5, 1990. pp. 27-34. ACM DL Reference
"Judgment-based Automatic Dialog Editor" Given a textual specification of just the
contents and their types, creates a dialog box Separately specify which look-and-feel
(not part of the specification) Defines mapping from types to widget
selection Graphic design rules for "nice" layout
10© 2013 - Brad Myers
DON (Won Chul Kim & Foley, InterCHI'93, pp. 430-437)
ACM DL Reference Ultimate in dialog box layout Kim's PhD thesis Works with OpenLook and devGuide Allows interactive designer guidance (preferences) on
sizes, layout, widget choice, etc. Can also choose among proposed layouts
Sophisticated 2-D layout Tries to balance dialog box Groupings of related items Effective use of white space (even margins, minimize wasted
space) Generates multiple designs and uses an evaluation metric to
choose.11© 2013 - Brad Myers
Don, pictures
PDF, with other pictures
12© 2013 - Brad Myers
Generating Full UI
These next tools require a specification of the full UI
Usually have rule-based components Specifications are in a special language
13© 2013 - Brad Myers
Interactive Transaction System (ITS)
Bennett, et.al., UIST'89 pp. 67-75 Wiecha, et.al. CHI'89, pp. 277-282 Wiecha, et.al., ACM TOIS, 8(3), Jul'90, pp. 204-236 Goal: capture designers knowledge as style rules
So unlike other systems, designer is required to edit the rules, not just the specification
All of UI must be created by editing the rules no interactive editing of generated interface (since then the knowledge
about why the generated interface wasn't good enough would be lost) Like dialog-box systems, separate specification of content
and style Style-independent tags associated with content "Style expert" programs the style for each tag Styles include both output (display) and input (interaction
techniques) specifications 14© 2013 - Brad Myers
ITS, cont. Can handle dialog boxes, forms, node-link diagrams,
kiosk frames, etc. Used for a number of internal IBM applications Used for all the information services at Expo'90 in Spain
Information, maps, restaurant reservations, etc. IBM researchers and content experts were in Spain for months
Evaluation + Full representation of design may increase re-use - Design specification ends up containing many specific "hacks"
used to achieve specific effects in single interfaces - Complex formal language for specification and rules
Pictures from ITS, TOIS, 8(3), pp. 213, 215, 217
15© 2013 - Brad Myers
© 2013 - Brad Myers 16
ITS, pictures
The User Interface Design Environment (UIDE)
Foley, et. al. CHI'88, pp. 67-72 Foley, et. al. IEEE Software, Jan'89, 25-32; Sukaviriya, et. al. InterCHI'93, pp. 375-382 Long-term project of Foley's at George
Washington and Georgia Tech Ended about 1994 when Foley left
A loose collection of separate implementations: IDL's transformations DON dialog boxes (described above) Sukaviriya's animated help Martin Frank's work (EET in Event-Based lecture) - etc.
17© 2013 - Brad Myers
UIDE Programmer defines Knowledge-base "schemas"
describing parts of the interface: Objects: in a class, sub-class inheritance hierarchy Actions: what can be done in the interface
Pre-conditions and post-conditions are in a very limited language counting, booleans, simple tests used for testing enabled and explaining why
Transformations change among equivalent UIs: e.g. Currently-selected obj <=> currently-selected cmd performed based on pre-, post-conditions
18© 2013 - Brad Myers
Pictures from UIDE
19© 2013 - Brad Myers
UIDE, cont. Automatic generate help for why commands are
not available Sukaviriya, et. al. AVI’94, Pages: 44 - 52 Animated help provides animations as a tutorial
Determines what needs to be done to demonstrate action Sequence of actions E.g. bring windows to the front, create an object, etc.
20© 2013 - Brad Myers
UIDE
Evaluation + Support for more than dialog boxes - Pre and post condition language is weak
can't express the test "if the selected object is a polygon..."
- Model language is a different, difficult language to learn
21© 2013 - Brad Myers
Humanoid
Szekely, et. al. UIST'90, pp. 1-9 Szekely, et. al. CHI'92, pp. 507-514 Szekely, et. al. InterCHI'93, pp. 383-390 High-level UIMS for Manufacturing
Applications Needing Organized Iterative Development
Model application data and interaction similar to UIDE
Model whole application: semantics + interface
22© 2013 - Brad Myers
Humanoid, cont. System picks generic interaction techniques
immediately using "templates" Designer can refine interface iteratively by
creating more specific sub-classes: Single-Command-Input-With-Alternatives Single-Command-Input-With-Few-Alternatives Allows exploration with incomplete designs
Evaluation + Much richer specification language than UIDE - More complex to define interfaces (more to learn)
but interactive tools help23© 2013 - Brad Myers
Humanoid Pictures
Pictures from Humanoid, CHI'93 pp. 384
24© 2013 - Brad Myers
MasterMind Neches, et. al. ACM 1993 Intelligent User Interfaces Workshop, pp.
63-70 Models Allowing Shared Tools and Explicit Representations to
Make Interfaces Natural to Develop Idea: combine UIDE and Humanoid Support entire life-cycle: early conceptual design through
maintenance Knowledge base is shared among all tools
Knowledge base serves as an integrating framework for various tools at design time and run time.
Spent a lot of time negotiating on how to combine models Lots of different parts to the model Personnel and coordination problems in doing the research Used Amulet!
25© 2013 - Brad Myers
MasterMindOne of the first system to integrate multiple models
together
26© 2013 - Brad Myers
27
MastermindPictures
© 2013 - Brad Myers
Research continued…
Fragmented into two different approaches Software engineering approach (early 90’s-)
Very detailed models to improve overall design process Intelligent design assistants instead of automatic generation Significant use of task models
Ubiquitous computing approach (2000-) Tons of “invisible” processors that perform tasks for us UIs for these processors are presented on other devices
(mobile phone, PDA, speech, etc.) Impossible to manually build user interfaces for every
combination
28© 2013 - Brad Myers
What are task models, anyway?
Key part of many current HCI approaches Description of the process a user takes to reach a
goal in a specific domain Typically have hierarchical structure
Introduced by GOMS
Number of different task modeling languages GOMS, CogTool UAN ConcurTaskTrees
29© 2013 - Brad Myers
ConcurTaskTrees Developed by Fabio Paterno et al.
for the design of user interfaces
Goals Graphical for easy interpretation Concurrent model for representing UI
tasks Different task types
Represent all tasks, including those performed by the system
30© 2013 - Brad Myers
Building/Editing Task Models
Tools are available ConcurTaskTrees
Environment
http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”
31© 2013 - Brad Myers
Software Engineering Approach Commercial work “Model-based design” Example: BPMN “business
process modeling notation” Business experts should be
able to author models Converted into code to
support the process (requires people) Keynote at ICSE’08: Herbert Hanselmann: Challenges in
Automotive Software Engineering “Model-based design (MBD) of functional behaviour has been a
big help in the recent past”
32© 2013 - Brad Myers
Mobi-D Angel Puerta, IUI’97 Set of tools to support
a clearly defined development cycle
Uses a series of different models
Explicit relationships that specify how the models are related to each other
Explicit interaction between end users and developers
33© 2013 - Brad Myers
XIMLeXtensible Interface Markup Language XIML.org Based on Mobi-D work Supports full development lifecycle Used by RedWhale Software to drive
their interface consultant business They have developed many tools
move interaction data to/from XIML Leverage data in XIML to better
understand various interfaces Automate parts of the interface design
process
34© 2013 - Brad Myers
Other Systems
UIML (http://www.harmonia.com/)
Originally a research project at Virginia Tech, now being developed commercially by Harmonia
Goal is platform independent language for describing UI Early versions were not very platform independent Recent versions using task models to automatically generate parts of the old
language that were not platform independent
Teresa (http://giove.cnuce.cnr.it/teresa.html)
Transformation Environment for inteRacti Tool for taking ConcurTaskTrees models, building an abstract interface, and then
building a concrete interface on multiple platforms.
USIXML (http://www.usixml.org)
Many of the same features of XIML Novel aspect is the use of graph structure for modeling relations (seems very
complex)35© 2013 - Brad Myers
Ubiquitous Computing Approach
“Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.”
-Dan Olsen (Xweb paper)
The web comes close to solving this problem, but is interactively insufficient.
36© 2013 - Brad Myers
Systems addressing UI issues
XWeb Now known as ICE – Interactive Computing Everywhere
ICrafter A system for integrating user interfaces from multiple
devices Supple
A system for automatically generating interfaces with a focus on customization/personalization.
Personal Universal Controller Jeff Nichol’s research…
37© 2013 - Brad Myers
XWeb
Work by Dan Olsen and group at BYU E.g. UIST’2000, pp.191 - 200
Premise: Apply the web metaphor to services in general Support higher levels of interactivity
38© 2013 - Brad Myers
XWeb Example
Interface
39© 2013 - Brad Myers
ICrafter
Part of the Interactive Workspaces research project at Stanford
Ponnekanti, et. al. Ubicomp’2001 Main objective:
“to allow users of interactive workspaces to flexibly interact with services”
Contribution An intelligent infrastructure to find services, aggregate
them into a single interface, and generate an interface for the aggregate service.
In practice, much of the interface generation is done by hand though automatic generation is supported.
40© 2013 - Brad Myers
Supple
Eventual goal is to support automatic personalization of user interfaces
Treats generation of interfaces as an optimization problem
Can take into account usage patterns in generation
Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal.
41© 2013 - Brad Myers
Modeling Users with Traces
Supple uses traces to keep a usage model Sequences of events:
<interface element, old value, new value>
Interfaces are rendered taking the traces into account (though traces are not required)
Trails are segmented at interface close or reset
42© 2013 - Brad Myers
Generating with Optimization Uses a branch-and-
bound search to explore space of alternatives Guaranteed to find
an optimal solution
43© 2013 - Brad Myers
Screenshots
44© 2013 - Brad Myers
Personal Universal Controller Jeff Nichol’s PhD work Problem:
Appliance interfaces are too complex and too idiosyncratic.
Solution: Separate the interface from the appliance and use
a device with a richer interface to control the appliance: PDA, mobile phone, etc.
Goal: Generate high-quality UIs45© 2013 - Brad Myers
Control
Feedback
Approach
Specifications
Appliances Mobile Devices
Use mobile devices to control all appliances in the environment
Key FeaturesTwo-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation
46© 2013 - Brad Myers
Properties of PUC Language State variables & commands
Each can have multiple labels Useful when not enough room
Typed variables Base types: Boolean, string,
enumerated, integers,fixed-point, floating-point, etc.
Optional labels for values Hierarchical Structure
Groups
47© 2013 - Brad Myers
Dependency Information Crucial for high-quality interfaces Expressed as <active-if> clauses
Operations: Equals, Less-Than,
Greater-Than Combined Logically
AND, OR Used for:
Dynamic graying out Layout Widget selection
48© 2013 - Brad Myers
Generating Consistent UIs
Personally consistent Reduce learning time Add unique functions
49© 2013 - Brad Myers
Generating Combined UIs
For multiple appliances, such as home theaters
Specify content flow Combined controls
50© 2013 - Brad Myers
Summative Study Compared PUC to manufacturer’s
interfaces for HP and Canon printer/fax/copiers
PUC twice as fast, 1/3 the errors Consistent: another factor of 2 faster
51© 2013 - Brad Myers0
200
400
600
800
1000
1200
1400
HP HP-PUC HP-Consistent
Canon Canon-PUC Canon-Consistent
Ave
rag
e T
ime
(sec
)
0
2
4
6
8
10
12
14
16
18
HP HP-PUC HP-Consistent
Canon Canon-PUC Canon-Consistent
Num
ber
of F
ailu
res
Model-based systems advantages/disadvantages + Separate specification of UI from content + Automatic reformatting, retargeting for different
platforms, customization to users + May allow programmers (non-experts) to write
specification and have a good UI automatically created But this didn’t really work out
– Result is often unpredictable – Often UI can be worse UI than hand-drawn – Sometimes model is larger than the code it would
replace – Model often in a different language that must be
learned52© 2013 - Brad Myers