Interaction techniques for post-WIMP interfaces

Lawrence SambrooksSupervisor: Dr Brett Wilkinson

2

“WIMP GUI’s based on the keyboard and mouse are the perfect interface only for creatures with a single eye, one or more single-jointed fingers, and no other sensory organs.” – Bill Buxton [1]

Summary

• Research question

• Background

• Methodology

• Proposed architecture

• Technology

• User evaluation

• Termination criteria

• Timeline

• Questions

3

What is post-WIMP?

4

Microsoft Surface

• Bimanual

• High bandwidth

• Intuitive

• Invisible

• Multi-finger

• Multi-sensory

• Multi-user

• Task focussed

Research question

Investigate and compare various interaction

techniques for post-WIMP interfaces

5

Background

• WIMP GUI’s were popularised by the Mac in 1984 and later copied to PC and Unix workstations [1]

• Input and output devices remain largely unchanged from what was demonstrated by Douglas Engelbart in 1968 [4]

• Physical input devices limit the interactions that can take place [3]

6

Methodology

• Develop a simple, multi-modal 3D data visualisation application with support for three interfaces:• Multi-touch• Gesture• Mouse and keyboard

• Provide the same functionality for each interface and allow the interactions to differentiate between them

• Test early and test often [2] 7

8

Methodology

SpaceStuntz (created by Martin Henschke)

• Utilise data sets provided by existing project [7]

The focus is HCI

9

HCIData

visualisation

Architecture

10

Mouse and keyboard interface Multi-touch interface Gesture interface

Data input, visualisation, core functionality, etc.

App

licat

ion

UI

11

Data visualisation

Technology

Hardware:• Windows 7 tablet PC

• Kinect

• Mouse and keyboard

Software:• Windows 7 SDK (multi-touch)

• Kinect SDK

• Visual Studio 2010 + XNA Game Studio 4.0

• TechCraft block engine [5]

12

Microsoft Kinect

User evaluation

Target groups:• Honours, Masters, and PhD students

• Academic staff

Feedback channels:• Surveys

• NASA TLX (Task Load Index) [6]

• Performance tests

• Internal program statistics

13

Termination criteria

• All three interfaces are functional

• A sufficient sample of data has been collected

for each of the interfaces such that a

reasonable conclusion can be reached

14

TimelineJul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

Literature review

Product development

Ethics approval

User evaluation

Thesis

Technology research

Expo

15

Conclusion

In summary:• Investigate three interaction techniques for a post-WIMP

interface

• Develop a simple 3D visualisation application

• Focus on the HCI aspects rather than data visualisation

• Collect feedback for each interface via user evaluation

More information:• http://wiki.csem.flinders.edu.au/bin/view/Projects/

ProjectSamb0014

16

Questions?

Hauptmann, A. G. Speech and gestures for graphic image manipulation. SIGCHI Bull., 20, SI 1989), 241-245.

Bolt, R. A. and Herranz, E. Two-handed gesture in multi-modal natural dialog. In Proceedings of the Proceedings of the 5th annual ACM symposium on User interface software and technology (Monteray, California, United States, 1992).

Thorisson, K. R., Koons, D. B. and Bolt, R. A. Multi-modal natural dialogue. In Proceedings of the Proceedings of the SIGCHI conference on Human factors in computing systems (Monterey, California, United States, 1992).

Nielsen, J. Noncommand user interfaces. Commun. ACM, 36, 4 1993), 83-99.

Jacob, R. J. K. Human-computer interaction: Input devices. ACM Comput. Surv., 28, 1 (Mar 1996), 177-179.

van Dam, A. Beyond WIMP. Computer Graphics and Applications, IEEE, 20, 1 2000), 50-51.

van Dam, A. User interfaces: disappearing, dissolving, and evolving. Commun. ACM, 44, 3 2001), 50-52.

[1] van Dam, A. Post-WIMP user interfaces: the human connection. Springer-Verlag London Ltd, Godalming, 2001.

Shen, C., Ryall, K., Forlines, C., Esenther, A., Vernier, F. D., Everitt, K., Wu, M., Wigdor, D., Morris, M. R., Hancock, M. and Tse, E. Informing the Design of Direct-Touch Tabletops. Computer Graphics and Applications, IEEE, 26, 5 2006), 36-46.

Aoyama, D. A., Hsiao, J. T. T., Cardenas, A. F. and Pon, R. K. TimeLine and visualization of multiple-data sets and the visualization querying challenge. J. Vis. Lang. Comput., 18, 1 (Feb 2007), 1-21.

Cawthon, N. and Moere, A. V. The effect of aesthetic on the usability of data visualization. IEEE Computer Soc, City, 2007.

Hornecker, E. "I don't understand it either, but it is cool:” visitor interactions with a multi-touch table in a museum. City, 2008.

Chan, A. T. S., Leong, H. V., and Kong, S. H. Real-time Tracking of Hand Gestures for Interactive Game Design. IEEE, New York, 2009.

Edelmann, J., Schilling, A. and Fleck, S. The DabR: A multitouch system for intuitive 3D scene navigation. City, 2009.

Petersen, N. and Stricker, D. Continuous natural user interface: Reducing the gap between real and digital world. City, 2009.

Reisman, J. L., Davidson, P. L. and Han, J. Y. A screen-space formulation for 2D and 3D direct manipulation. In Proceedings of the Proceedings of the 22nd annual ACM symposium on User interface software and technology (Victoria, BC, Canada, 2009).

[2] Yee, W. Potential Limitations of Multi-touch Gesture Vocabulary: Differentiation, Adoption, Fatigue. Springer-Verlag Berlin, City, 2009.

[3] Andrews, D. and Teoh, S. T. MTVis: Tree Exploration Using a Multi-Touch Interface. Spie-Int Soc Optical Engineering, City, 2010.

[4] Lee, J. C. In search of a natural gesture. XRDS, 16, 4 2010), 9-12.

Knoedel, S. and Hachet, M. Multi-touch RST in 2D and 3D spaces: Studying the impact of directness on user performance. City, 2011.

[5] TechCraft block engine: http://techcraft.codeplex.com

[6] NASA TLX: http://human-factors.arc.nasa.gov/groups/TLX

[7] Hobbs, D. Rehabilitation gaming for children with Cerebral Palsy (ongoing PhD

research).

References