Interaction Techniques with Mobile Devices Jingtao Wang [email protected] March 6th, 2006...

Interaction Techniques with Mobile Devices

Jingtao Wang

[email protected]

March 6th, 2006 Guest Lecture for CS160

Agenda

Why Mobile Devices MattersUbiquitous ComputingKey Challenges in Designing Mobile

ApplicationsInput Techniques for Mobile DevicesOutput Techniques for Mobile DevicesInteract With Other Devices

Why Mobile Devices Matters

6.5 billion people in the world 1.5 billion cell phones worldwide 500 million PCs (?) 46 million PDAs 1 million TabletPCs

Challenge: How can handheld devices improve the user interfaces of everything else, and not just be another gadget to be learned

6,500

1,500

50046 1

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

Peopl

e

Cell P

hones PCs

PDAs

Table

tPCs

Mill

ion

s

Agenda



Mark Weiser (1952 – 1999)

Introduced the idea of “ubiquitous computing”

Weiser’s Vision

“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it”

Weiser’s Vision: Pervasive

Mainframemany people share 1 computer

PC1 person with 1 computer

Ubicompmany computers server each person

Agenda



Key Challenges in Making Mobile Applications

Limited Physical Resources CPU, Memory, Screen Size, Input Devices,

Battery Life etcDiversified Context of UseDifferent ActivitiesLimited Attention

Limited Physical Resources

A mobile device usually has 1/100 CPU power, 1/30 Screen resources, 1/20 Memory, and extremely limited input devices when compared with desktops in the same era.

Small Screen Geography is different

a. Large Screenb. Small Screen

Diversified Context of Use

Different Activities

People use small-screen devices for different activities than desktops; don’t assume you understand these activities already

Limited Attention

Don’t assume your applications have people’s full attention; they’re doing something else while using your device.

Context, Activity, Attention

There is more opportunity for purpose-specific or context-specific devices than for general-purpose solutions that try to work for everyone in any situation.

One Sentence Summary

Mobilize, Don’t Miniaturize !

There is no silver bullet in designing mobile applications, but there is one sentence you should remember -

Agenda


ApplicationsInput Techniques for Mobile DevicesOutput Techniques for Mobile Devices

Input Techniques for Mobile Devices

PointingText Input

(Virtual) Keyboard InputHandwriting InputSpeech Input

Marker Based Input

Common Pointing/Navigation Techniques

iPod Dialpad

TrackPoint

JogDial

Touch ScreenFour-directional keypad

TinyMotion – Camera Phone Based Pointing

Detecting the movements of cell phones in real time by analyzing image sequences captured by the built-in camera.

Typical movements include - horizontal and vertical movements, rotational movements and tilt movements.


PointingSensor Augmented Input

Text Input(Virtual) Keyboard InputHandwriting InputSpeech Input

Marker Based Input

(Virtual) Keyboard Input

How to Make QWERTY Keyboards Portable ?

Break

Making QWERTY Keyboards Portable

Reducing the Absolute SizeReducing the Number of KeysMaking Keyboards FoldableProjective Keyboard

Projective Keyboard

From http://www.vkb-tech.com

Projective Keyboard – Working Mechanisms

1. Template creation

2. Reference plane illumination

3. Map reflection coordinates

4. Interpretation and

communication

Can We Perform Better Than QWERTY?

Originally QWERTY layout is manually optimized for two handed, alternative typing to minimize mechanical jam

OPTI ATOMIK

OPTI II FITALY

Theories Behind Quantitative Keyboard Layout Optimization

Fitt’s LawDigraph Distribution Model in a Language

Can We Use Less Buttons than a Full QWERTY?

12-button Keypad15-button Keyboard

Half Keyboard

Disambiguation Methods for Reduced Size Keyboard

The QWERTY keyboard itself is ambiguous! ( A vs. a, 3 vs. #)

Pressing Several Keys together (shift key) Multiple Key Press

Multi-Tap (22.5 wpm*) Two-Key Input (25.0wpm*)

Dictionary/Statistics Based Disambiguation Methods T9/T15 (45.7 wpm*) LetterWise

*Performance Upper Bound Estimation from Silfverberg 2000

FingerSense – Button Disambiguation by Fingertip Identification

Differentiating a pressing action by identifying the actual finger involved

Can be Faster than Regular Tapping When the Adjacent Tapping Involves Different Fingers and Different Buttons (59% on a phone keypad)




Marker Based Input

Handwriting Input

1938 George Hansel, U.S. Patent 2,143,875, machine recognition of handwriting

1957 T. L. Dimond's stylator - the first on-line handwriting recognizer prototype

Newton, Palm Pilot, PocketPC, CrossPad, TabletPC

Handwriting Recognition - Terminology

Printed Character Recognition (OCR) Relatively mature these days, key challenges – layout

analysis, fonts recovery, robust recognition for low quality, low resolution input

Major Usage – Digital Library Handwritten Character Recognition

Online HWR (With Temporal info) Character, Word, Sentence Level

Offline HWR (Using raster image as input, no temporal info)

Major Usage : Bank Check Recognition, Postal Automation

Word/Sentence Level Recognizer

Build on Top of Character Recognizer General Strategy

Over Segmentation Call Character/Component Recognizer, Get a List

of Candidates with Scores Apply Geometry Spatial Information ( size,

component gap ) Language Information (Dictionary, Language Model etc) to Each Sub Path

Use Hypnosis Search (Dynamic Programming, A* etc) to Determine the Best Possible Path

Challenges in Online HandwritingRecognition

Character Set/Dictionary Size (Especially Asian Languages!)

Cursive Writing Styles/Broken Strokes/Duplicate Strokes/Omitted Components

Stroke Order VariationsLimited memory and CPU Power in Small

Devices

Some Prototype Recognizers from IBM

New Form Factors - Anoto Pen

Commercial Product is Available In the U.S. Market (Logitech IO Pen)

Uses A Camera Mounted Beside the Tip of the Pen and Preprinted Dot Patterns to Detect Pen Movment

SHARK – Shorthand Writing on Stylus Keyboard

A Combination of Virtual Keyboard and Handwriting Recognition

Writing Shape of a Word (Shorthand) is Defined By the on Screen Location of Characters in the Word

EdgeWrite Input

An EdgeWrite user enters text by traversing the edges and diagonals of a square hole imposed over the usual text input area

Faster and More Reliable Than Regular Graffiti

Especial Useful for People with Motor and Muscle Disabilities




Marker Based Input



Text Input (Virtual) Keyboard InputHandwriting InputSpeech Input

Marker Based Input

Emerging Marker Based Interactions on Camera Phones

Towards More Sensitive Mobile Devices

Agenda



Peephole Displays (With Demo)

Zoomable Interface on Mobile Devices

ZoneZoom By Microsoft Take advantage of spatial

memory

VS.

Halo - A Virtual Periphery for Mobile Devices

Provding Visual Cue for Objects Located Out of the Small Screen

Agenda



Using Mobile Devices with Desktop Computers

Pebbles Project at CMU Using a PDA as additional keypad, touch pad, scroll

wheel and controller of PointPoint slides for desktop Applications

http://www.pebbles.hcii.cmu.edu/

Using Mobile Devices with Laptops

Wang 2002

Using Mobile Devices with Large Displays

Ballagas 2005

Question and Answer

Backup Slides

Electromyographic (EMG) Keyboard

NASA Virtual Keyboard SenseBoardKeyboard

The General Flow of Handwriting Recognition

Major Players in this Area (Embedded)

English ART - ART Recognizer CIT - Jot IBM - Derived from Multi-lingual version Microsoft - Transcriber ( Licensed version of Calligrapher) & Own Sin

gle character recognizer Motorola Paragraph - Calligrapher

Chinese/Japanese FineArt - GoGoPen Hanwang - more than 70% PDA market share in mainland China IBM Embedded HWR Motorola Lexicus - DragonPen PenPower - most influencial in Taiwan

Sensor Augmented Pointing

Interaction Techniques with Mobile Devices Jingtao Wang [email protected] March 6th, 2006...

Documents

Transcript of Interaction Techniques with Mobile Devices Jingtao Wang [email protected] March 6th, 2006...