1

The Future Of Human-Computer Input Interfaces

Stephanie BudimanJohnny Cham

Karthik NandakumarOsbert Poniman

Mauhay Mary Esther Samson

2

Computing Anytime Anywhere!

Motivation: In today’s world, easy access to information and computing is required anytime and anywhere

3

HCI

Human-Computer Interface (HCI)

• HCI is the technology that connects man and machine

• Robust HCIs are needed to enable ubiquitous computing

We focus only on input interfaces in this presentation

Human Computer

Thoughts

ActionInput

InterfaceAction

Recognition

Task Execution

Understanding

Output Interface

Sensory Perception

Rendering

4

Technology Paradigms for Input Interfaces

Graphical User Interfaces

Command Line Interfaces

Batch Interfaces

Natural User Interfaces Neural Interfaces

5

Value Proposition

Improvements in component technologies

AccuracyThroughputAffordability

User ExperienceEase of UseSociabilityMobility

CLI

GUI

Natural UI

Neural Interfaces

Low Medium HighLow

Medium

High

6

Key Components of Input Interfaces Human-Computer Input Interfaces

Neural Interfaces

Natural UI

Speech

Micro phone

Neural electrodes/

sensors

GestureTouch

3D Camera

Touch sensor

Tracking & Recognition

Software

Materials/Nanotechnology

Signal Processing Hardware (Semicon-ductors)

7

Speech Interfaces

Key Components

Microphone

Automated Speech Recognition (ASR) and Natural Language Understanding (NLU) Software

Possible methods of improvement are

• Increase Signal to Noise Ratio (SNR) from microphone

• Achieve human-level performance on ASR/NLU tasks

Key value that needs improvement is Accuracy

8

Microphone Technology

Electret Condenser Microphone (ECM)

SNR: 55-58 dB

MEMS Digital Microphone

SNR: 61 dBFlatter frequency response

Smaller size (CMOS fabrication)

Noise levels in a microphone is close to the thermal noise limit Analog Devices, MEMS Microphone Technology, October 2010

9

Microphone: Method of ImprovementMicrophone array can mitigate background noise & interference1

1. LOUD project from MIT Computer Science and Artificial Intelligence Laboratory, 2005

2. Microphone Array project in MSR: Approach and Results, Microsoft Research, June 2004

Noise suppression algorithms can increase SNR by 18dB with just 4 microphones in an array2

Improvement in SNR Corresponding decrease in Word Error Rate

10

Automated Speech Recognition (ASR)

“Increase in vocabulary sizes needs exponential increase in computing power due to potential combinatorial explosions”

L. Rabiner, “Challenges in Speech Recognition”, NSF Symp. on Next Gen. ASR, 2003

11

ASR Accuracy Improvements

ASR accuracy is acceptable only in some niche applications

WO

RD

ER

RO

R R

ATE (

%)

NIS

T B

ench

mark Te

st Histo

ry – M

ay 2

009

YEAR

12

ASR Accuracy in Text Dictation

• Stand-alone speech interfaces may be useful for tasks like dictation

• Speech as an important modality in multimodal user interfaces (e.g., Microsoft Kinect) may be the future

* http://blogs.msdn.com/b/sprague/archive/2004/10/22/246506.aspx

*

13

Touch Interfaces

Key Component

• Touch screen

Microsoft SurfaceiPad Touch

Key values that need improvement are Throughput and Affordability

14

Comparing Touch UI with Mouse• Fitt’s Law: Time required to move to a target area displayed

on screen is proportional to index of difficulty (distance to the target/width of the target area)

Touch UI has better throughput than mouse for more difficult tasks

Muller, L.Y.L., “Multi-touch displays: design, applications & performance evaluation”, 2008

15

Improving Throughput of Touch UI

B. Buxton, “Multi-Touch Systems”, Microsoft Research, 2007

Single Touch• 1960, IBM• 1972, Plato IV Computer• 1992, Simon (IBM):1st smart

phone

Single Touch with Pen Input• 1991, Digital Desk• 1996, Palm Pilot• Pen writes, touch

manipulates

Multi-touch• 2001, Fingerworks• 2005, Apple

Interaction based on large multi-touch screens can increase the throughput of touch user interfaces

16

Touch Screen Technologies

Infrared (IR)

Resistive

Surface Acoustic Wave

Surface Capacitive

Source: 3M Touch System Website

17

Comparison of Touch TechnologiesTouch

technology Capacitive Resistive InfraredSurface

Acoustic Wave

Transparence Very good 90%

75%-85% Very good 90%

Very good 90%

Resolution Good Good Limited due to spacing of IR

sensors

Good

Display size Up to 21” Up to19” Up to 60” Up to 30”

Touch method Human touch Can use any pointing device

Can use any pointing device

Finger, gloved, hand or soft tip

Touch Reliability

Unlimited 50 million touches

Unlimited 100 million touches

Cost (15”Monitor) $680 $695 $1356 $820

Source: Frost & Sullivan, Advances in Haptics & Touch Technology, 2010

18

Future of Touch: Tangible Bits

Manipulating digital objects through physical objects

19

Gesture Interfaces

Key Components

3D Camera (image sensor)

Tracking, Recognition &

Gesture Understanding

Software

Key values that need improvement are Accuracy, Throughput and Affordability

20

Gesture UI: Methods of ImprovementRequired improvements in image sensor technology

Accuracy

• Higher spatial resolution (number of pixels)

• Robustness to lighting changes (high pixel sensitivity)

• More accurate depth sensing (lower depth error)

Throughput

• Higher frame rate (temporal resolution)

Affordability

• Smaller pixel size reduces price per pixel

21

Image Sensor Technologies

CMOS-based image sensors are also expected to follow Moore’s Law in size and cost scaling

T. Suzuki, “Challenges of Image-Sensor Development”, ISSCC, 2010

As pixel size decreases, resolution improves (more pixels per area), but sensitivity decreases because the “light available per pixel” will become less

Back illuminated CMOS technology provides better trade-off between pixel size and sensitivity than traditional charge coupled device (CCD)-based image sensors

22

Camera Technology Improvements

T. Suzuki, “Challenges of Image-Sensor Development”, ISSCC, 2010

Reducing pixel-size (green square) and improving sensitivity (Yellow circle ) miniaturized cameras without reducing quality

23

Camera Price Improvements

K. Wiley, “Digital Photography”, www.keithwiley.com

Number of pixels (resolution) has increased, while price per pixel has decreased

Year

Pric

e pe

r pi

xel f

or a

12

MP

cam

era

24* R. Lange, “3D Time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology”, PhD Thesis, 2000

3D Image Sensor Technology• Time of flight (ToF)

cameras require high temporal resolution* (70 picoseconds for 1cm depth resolution)

• CMOS technology now has the required temporal resolution to enable production of affordable ToF 3D cameras

• 3D cameras will improve the accuracy of gesture UI

Cost-effective 3D image sensors are now becoming available (e.g., Microsoft Kinect ~ 150 USD)

Dep

th E

rro

r/D

ista

nce

Distance to image

25

Neural InterfacesKey Component

Brain scanning device

Key values that need improvement are Accuracy, Throughput and Affordability

Required improvements in brain scanning technology

Accuracy – Higher spatial resolution

Throughput – Higher temporal resolution

Affordability – Size and better materials

26

SPECT

EEG

1936 1950 1972 19751968

CT Scan

1983

MEG

1991

fMRINIRS

1973

MRI PET

US$2.9M

US$5M-7M US$1M-1.5M

US$250K US$2.4M US$0.5M-3M

US$180K- 250K>US$30K

Baranga, A. B.-A. (2010). "Brain's Magnetic Field: a Narrow Window to Brain's Activity". Electromagnetic field and the human body workshop, (pp. 3-4).

Key Brain Scanning Technologies

Electro Encephalo Graphy

Magneto Encephalo Graphy

27

Comparison of Technologies

• Ideally, a non-invasive technology with high spatial resolution and high temporal resolution is required

• Additionally, the technology must be affordable and portable in order to be useful in HCI applications

Gerven, M. v., et al., “The Brain-Computer Interface Cycle”, J. Neural Eng, 2009

Invasive

Non-invasive

Neuron can fire ~0.1mm (spatial) & ~10 ms (temporal)

28

Spatial Resolution Improvement

While spatial resolution is important for accuracy, high temporal resolution is critical for user interfaces

R. Kurzweil, “The Singularity is Near”, 2005

29

ElectroEncephaloGraphy (EEG)

• Non-invasive interface using electrodes to pick up brain signals

• Key limitation: Poor spatial resolution

Increasing number of EEG electrodes may provide limited improvement in spatial resolution and higher SNR

J. Malmivuo, “Comparison of the Properties of EEG and MEG”, Intl J of Bioelectromagnetism , 6 (1), 2004

30

MagnetoEncephaloGraphy (MEG)

Baranga, A. B.-A. (2010). "Brain's Magnetic Field: a Narrow Window to Brain's Activity". Electromagnetic field and the human body workshop, (pp. 12).

fT

31

MEG: Improvements in Millimeter-scale Atomic Magnetometer

Target: ~100fT and <100Hz

J. Kitching, et al., “Uncooled, Millimeter-Scale Atomic Magnetometers”, IEEE Sensors 2009 Conference (pp. 1844-1846)

A: 2004B: 2007C: 2007D: 2009

32

Further Scope for Improvement• Hybrid technologies

– Enable high levels of accuracy in diagnosis that individual modalities cannot offer.

• E.g: MEG, EEG and MRI1 or PET/CT, PET/MRI2

• Better shielding for noise reduction

– Low frequency noise: use flux-entrapment shields

– High frequency noise: use lossy magnetic shields based on induced eddy currents

• Reduce costs

– Open-source: OpenEEG

[1] Baranga, A. B.-A. (2010). "Brain's Magnetic Field: a Narrow Window to Brain's Activity". Electromagnetic field and the human body workshop, (pp.15).

[2] Stommen, J. (2011, Mar 25). "Superior capabilities boost hybrid imaging, says report“, Medical Device Daily

33

Opportunities in Input InterfacesMicrophone

Multitouch Sensors

3D camera & motion sensors

Haptic Devices

Neural Sensors

Speaking

Touch Events/

Gestures

Body/head movements

Free hand gestures

Facial expression

Eye gaze

Hand pressure

Brain activity

Speech Recognition &

Language Understanding

Touch/Gesture Recognition & Understanding

Neural Signal Analysis &

Understanding

User Interface System Design & Integration

Multimodal Fusion

Human Factors

Engineering

Context Aware

Services

34

Market Segments for Input Interfaces

Natural & Neural User Interfaces

SmartphonesTablet/

Wearable PCs

Consumer Electronics

TelevisionGames

Virtual Reality

Entertainment

Healthcare

ProstheticsTele-medicine Military

E-commerce

Security

AircraftAutomotive

Transportation

Tele-control

Robotics

Virtual / Collaborative

Learning

EducationGroup InterfaceTelepathic System

Telecommunications

35

Conclusions

• Ubiquitous computing requires new HCI paradigms

• Natural and Neural Interfaces are the future of human-computer input interfaces

• Touch interfaces have already diffused into the mainstream; speech and gesture interfaces are becoming more accurate and affordable

• Neural interfaces requires development of more accurate, cheap, and portable sensors

• Numerous entrepreneurial opportunities are available both in technology development & customization

36

THANK YOU!

Q&A?