1

Post-Wimp Interaction With Desktop Computers

Pierre Dragicevic

New Tools and Techniques

2

WIMP Interaction

n Standard Interaction Techniques:

n Windows, widgets, iconn Text input, click, double-click, d&d

+

n WIMP = Windows, Icons, Menus, Pointer

n Standard Devices:

n Mousen Keyboard

3

What Next and Why?

n WIMP system(Xerox Star, 1979)

4

Post-WIMP Interaction Paradigms

Better exploit our motor abilities:

n Bimanual Interaction

Map Zooming [Hinckley et al 98]

SmartSkin [Rekimoto 2002]

n « Multifinger » Interaction

5

Post-WIMP Interaction Paradigms

Better exploit our perceptual abilities:

n Information visualization

TreeMaps [Schneiderman et al]

Multiple screens & large displays

n Enhanced displays

6

Post-WIMP Interaction Paradigms

Make interaction more « direct »:

n Gestural interaction

Toolglasses [Bier et al 93]

n See-through tools

7

Two Research Directions

n Explore Post-WIMP interaction further:

n Propose innovative solutions to known problemsn Investigate alternative interaction paradigms

n Disseminate existing research:

n Integrate Post-WIMP paradigms into GUI toolkitsn Provide new design guidelines

8

My Research

n Explore Post-WIMP interaction further:

n Propose innovative solutions to known problemsn Investigate new interaction paradigms

n Disseminate existing research:

n Integrate Post-WIMP paradigms into GUI toolkitsn Provide new design guidelines

n Support the exploration of new approaches:

n Study more usable and more flexible toolsn Advocate a vision of Post-WIMP with sound examples

9

My Research

New tools

New techniques

10

New techniques

Introduction

Conclusion

II. Artistic Resizing

IV. Fold n’ Drop

I. Input Configurator

III. SpiraClock

New tools

11

I. Input Configurator

Pierre Dragicevic

Jean-Daniel Fekete

An Input Interaction Model forHighly Configurable Multi-Device

Interactive Systems

Introduction

Conclusion

II. Artistic Resizing

IV. Fold n’ Drop

I. Input Configurator

III. SpiraClock

12

Goal: Support Alternative Input

n WIMP system(Xerox Star, 1979)

n Input-Open System

13

Why Using Alternative Input?

the task the user the environnement

n Augmenting the bandwith

n Optimizing interaction according to:

14

Current State of the Art

n Very poor support for alternative input devices in:

n Current operating systemsn Current applicationsn Current toolkits

n Little research on input management

n Post-WIMP tools are specialized towards:

n A fixed set of input devicesn A fixed set of interaction techniques

n No approach deals with:

n Multiple and heterogeneous input (enriched / impoverished)n Flexibility and configurability

15

The Input Configuration Model

n Approach:

n Fully separate physical devices from applications

n Provide a flexible way to map devices to applications

16

The Input Configuration Model

n Device:

n Parametrizable black boxn Processes input into output

n I/O Slots:

n Simple typesn + compound types

n Implicit I/O:

n User inputn Feedback

Inputslots

Outputslots

Implicit input

Implicit output

17

The Input Configuration Model

n Device types:n System

(hardware ressources)n Adapter

(software processes)n Application

(specific to each app)

n Input Configuration:n A set of system and

application devicesconnected throughadapter devices

System devices

Application devices

Adapter devices

Input Configuration

18

The ICON Toolkit

n System Devices:

19

The ICON Toolkit

n Adapter devices:

n Data processing

20

The ICON Toolkit

n Adapter devices:

n Data processingn Graphical feedback

21

The ICON Toolkit

n Adapter devices:

n Data processingn Graphical feedbackn Interaction techniques

22

The ICON Toolkit

n Application Devices:n Example: IConDraw tools

23

The ICON Toolkit

n Graphical Toolkit Devices:

n Example: Swing devices

24

The ICON Toolkit

n The input configuration editor:

Availabledevices

Availabledevices

Editingpane

Editingpane

25

The ICON Toolkit

n The input configuration editor

26

The ICON Toolkit

n The input configuration editorn Dynamic wire drawing algorithm

27

The ICON Toolkit

n The input configuration editorn Fluid expand/collapse

28

The ICON Toolkit

n The input configuration editorn Fisheye-drag

29

Examples

1. Exploiting rich input

n Pressure-sensitive drawing inside IConDraw

30

1. Exploiting rich input

Examples

31

Examples

1. Exploiting rich input

n Symmetric bimanual interaction in IConDraw

32

Examples

1. Exploiting rich input

33

Examples

1. Exploiting rich input

n Integral control of the Responsive Face [Perlin 97]

34

Examples

1. Exploiting rich input

n Integral control of the Responsive Face [Perlin 97]

35

Examples

1. Exploiting rich input

36

Examples

2. Improving Accessibility

n Keyboard ? Pointing

2nd order control

37

Examples

2. Improving Accessibility

n Pointer ? Text editing

« Floating QuikWriting » [Perlin]

38

Examples

2. Improving Accessibility

n Pointer ? Text editing

« Floating QuikWriting » [Perlin]

39

Examples

3. Augmenting WIMP applications

n « Augmented pointing »

Filter Events

40

Examples

3. Augmenting WIMP applications

n « Augmented pointing » : smoothed cursor

41

Examples

3. Augmenting WIMP applications

n « Augmented pointing » : throwable cursor

42

Examples

3. Augmenting WIMP applications

n « Augmented pointing » : speech cursor

rightrightNoise…Noise… downdown

stopstoppresspressupupslowerslowerreleaserelease

43

Examples

3. Augmenting WIMP applications

n « Generalized Drag and Drop »

44

Projects using ICON

n Marina II (2001)Photograph-based architectural modelling tool

n Svalabard [Huot et al 2003]Post-WIMP sketching-based 3D modelling interface

n MaggLite [Huot et al 2004]Post-WIMP UIMS based on ICON

n LRI, University of Orsay [Appert et al 2003]Experimental comparison of WIMP / Post-WIMP techniques

n Smart Home and Impairment (INT/ENST/EDF 2005)Configurable software architecture for accessibility

n Student projects

45

II. Artistic Resizing

Pierre Dragicevic

Stéphane Chatty

David Thevenin

Jean-Luc Vinot

A Technique for RichScale-Sensitive Vector Graphics

Introduction

Conclusion

II. Artistic Resizing

IV. Fold n’ Drop

I. Input Configurator

III. SpiraClock

46

n Graphic designers are given more and more importance in GUIs

n Exploitation of vector graphics(e.g, Scalable Vector Graphics)

n IntuiKit [Chatty et al 2004]

Graphically Rich User Interfacs

47

IntuiKit: Example of Team Work

Low-fi prototyping

Graphic design

Programming

Integration

48

The Resizing Problem

n Fixedsize

n Naivescaling

n Common resizing

49

Expressing Resizing

n Resizing is commonly described using formulae

n These formulae are:

n Translated into code by the programmern Or used as an input to constraint-solving systems

w

h

wL

hL

yL

xLr

hB

wB

• xL = (w-wL) / 2• yL = (h-hL) / 2• wL = 20• hL = 10

• wB = 5• hB = 5

• r = 20

50

Expressing Resizing

n But designers think visually

n They are used to produce visual variants

n Variants are also useful to convey general laws

On whitepaper

On blackpaper

On redpaper

Mediumsize

Smallsize

51

Artistic Resizing

1. Designers produce variantsusing their authoring tool

2. IntuiKit interpretsthe example set

52

Artistic ResizingA scenario

53

Artistic ResizingA scenario

54

Artistic ResizingA scenario

55

Artistic ResizingA scenario

56

Artistic ResizingA scenario

57

Artistic ResizingHow does it work?

n Based on local interpolation of transformations

T1 T1’

58

Artistic ResizingHow does it work?

n Each variant of T1 is associated withthe example’s bounding box

T1 T1’

?

T1’’ T1’’’

59

n Orthogonal Interpolation hypothesis:

n Width only impacts the X contributionn Height only impacts the Y contribution

Artistic ResizingOrthogonal Interpolation

a11 a12 a13

a21 a22 a23

0 0 1

X

Y

1

X0

Y0

1

=

width

height

60

n Piecewise linear interpolations are performed along width and height separately:

Artistic ResizingOrthogonal Interpolation

width

height

a11a12 a13

a21a22

a23

61

Artistic ResizingProperties of Orthogonal Interpolation

n Preserves algebraic measures:

n If horizontal or vertical distances are the same on the examples they will remain the same

62

Artistic ResizingProperties of Orthogonal Interpolation

n Preserves relative ratios:

n Ratios that are invariant on the exampleswill remain invariant

63

Artistic ResizingProperties of Orthogonal Interpolation

n Preserves contacts:

n Two points that coincide on the exampleswill always coincide

64

Artistic ResizingProperties of Orthogonal Interpolation

n Preserves parallelism:

n Two lines that are parallel on the exampleswill remain parallel

65

Artistic Resizing

n Supports invariants but also finely tuned behaviors

w

h

wL

hL

yL

xLr

hB

wB• xL = (w-wL) / 2• yL = (h-hL) / 2• wL = 20• hL = 10

• wB = 5• hB = 5

• r = 20

66

Artistic ResizingBlending with traditional layout managers

n Containment hierarchies:

67

Artistic ResizingBlending with traditional layout managers

n Inside an « aqua-dock » layout manager:

68

n Demonstrational layout specification systems:

n Peridot [Myers 90]

n Chimera [Kurlander & Feiner 93]

n Constraint inference:

n Extensive searchfor invariants

n Sensitive to combi-natorial explosion

n Over-constrained orunintended rules

Artistic Resizingvs. Constraint Inference

n Orthogonal interpolation:

n Ensures the preservation of « interesting » invariants

n Fast computation, even with complex vector graphics

n Limitations are known, results are easy to predict

69

III. SpiraClock

Pierre Dragicevic

Stéphane Huot

Introduction

Conclusion

II. Artistic Resizing

IV. Fold n’ Drop

I. Input Configurator

A Continuous and Non-IntrusiveTechnique for Upcoming EventsIII. SpiraClock

70

Time Management Tools

n Calendars displays:n Good for long-term time

managementn But useless for

upcoming events

n Reminders and Alarms:n Prevent forgettingn But too intrusiven Don’t allow

continuousanticipation

71

SpiraClock

n Analog desktop clock

n Events are displayedas sectors

72

SpiraClock

n Event are aligned withminute marks

n Relative times can beread directly

73

SpiraClock

n One example: viewing bus departures

It’s 18h50.What busescan I catch ?

74

SpiraClock

n One example: viewing bus departures

75

SpiraClock

n One example: viewing bus departures

76

Uses of SpiraClock

n BusClock used in Ecole des Mines de NantesSince 2002

n Other applications:

n ShuttleClock - shuttles of University of Marylandn OutClock - synchronized with MS Outlookn CalendarClock - uses VCal exchange format

77

IV. Fold n’ Drop

Combining Paper-Based and

Crossing-Based Interaction Paradigms

for Dragging and Dropping betweenOverlapping Windows

Pierre Dragicevic

Introduction

Conclusion

II. Artistic Resizing

IV. Fold n’ Drop

I. Input Configurator

III. SpiraClock

78

Drag & Drop Scenario

n Moving a file into a folder

79

Drag & Drop Scenario

n Problem: the target is hidden

80

drag & drop

Drag & Drop Scenario

n Solution 1: adjust windows

adjust

81

cut

bring to the front

Drag & Drop Scenario

n Solution 2: cut & paste

82

paste

Drag & Drop Scenario

n Solution 2: cut & paste

83

Using Window Navigation

n Dual representation → Mapping problems (XP taskbar)

n Activation costs (mouse-over timeout on Exposé)

84

Using Window Navigation

n Activation is a superfluous concept for D&D tasks

85

The Fold n’ Drop Technique

n Folding a window

n The mouse leaves the window

n The mouse leaves then returns back

86

The Fold n’ Drop Technique

n Other gestures

n Leafing through

n Discarding a window

n Unfolding

87

Video

88

Fold n’ Drop builds upon…

n Paper-based window metaphors

n The « Folded » effect [Beaudouin-Lafon 01]

Ametista Toolkit [Roussel 03] AttrActive Windows [Denoue 03]

89

Fold n’ Drop builds upon…

n Gestural interfaces

n Natural, « physical » gestures

n Do not need a recognition engine

n Naturally blend with direct manipulation[see e.g., Pook et al 00]

n Little interference with other interactions

90

Fold n’ Drop builds upon…

n The crossing-based interaction paradigm[Accot & Zhai 02]

n Fn’D uses double-crossing and n-crossing

n Extends the paradigm:

• Point & Click → Cross boundaries• Drag → Push boundaries• Double-Click → Timed double-crossing

n Timing ismade visible:

91

Conclusion

Introduction

Conclusion

II. Artistic Resizing

IV. Fold n’ Drop

I. Input Configurator

III. SpiraClock

92

Conclusion

n Common desktop computing activities canstill be improved:

n Management of multiple tasks• Task switching• Data transfer between applications

n Management of personal data• Viewing, searching• Organizing

n Time management• Long-term• Short-term

n But how to design Post-WIMP techniques ?

93

ConclusionPost-WIMP GUI Design as a Creative Activity

n The success of a technique is difficult to predict:

n Efficiency + pleasingness

? Importance of « trial and error » (iterative design)

n But techniques also require fine tuning:

n Timings, input filtering, graphical effects, color design,…

? Need for closer experimentation-evaluation loops

n « Creative tools » allow to explore and test by ourselves:

n Flexible, openn High degree of control, finely tunablen Fast experimentation-evaluation loopn Simple and predictable

94

Conclusion5 characteristics of Post-WIMP Techniques

1. Specificity

2. High I/O bandwidthn Multiple input channelsn Advanced output devices or rich graphics

3. Directnessn Stimulus/Response compatibility

4. Reactivity and Fluidityn Responsive and continuous feedback

5. Straightforwardnessn Familiar metaphorsn Simple, predictable system behavior

95

ConclusionAbout Predictability

n Example: gestures in Fold n’ Drop

« I missed the fold »

« I returned backtoo late »

96

ConclusionAbout Predictability

n Another example: freehand polyline drawing

n Using a smart tool (stroke beautifier):

97

ConclusionAbout Predictability

n Another example: freehand polyline drawing

n Using a smart tool (stroke beautifier):

98

ConclusionAbout Predictability

n Another example: freehand polyline drawing

n Using a dumb tool (spring-loaded cursor):

99

ConclusionFuture Work

n Continue to explore new interaction paradigmsn Desktopn Non-desktop

n Generalize and theorize:n Intelligence vs predictabilityn Explicit gesturesn Augmented WIMPn …

n Papers, videos and demos:http://www.intuilab.com/~dragice