How to say it to your Smart Glasses?

Martin Neumann, innospiring GmbH iot conference Berlin 2015

about martin inventor

bloggerauthor

teacher

entrepreneur

legacy

chemist

what to expect?

• entertainment • new and relevant information • inspiration

no wannabe products, serious prototypes, hover boards or alike

the age of

SMARTCar

CityTV

Dust

HomeMeter

Bike

Robots

jash?*

*just another smart hype?

you are here

market - status quo

B2B or B2C?

• no AR end consumer product* or market yet • a lot of demo applications • main issues are price (~700 €), availability of

applications and content, use cases, familiarity, social acceptance

*besides universe2go

key features?!

key features?!

Something VERY important is missing here…

user interaction!

user interaction?voice

gesture

joystick ringbracelet

smartphonewatch

head/eye cursor

with wearables

user interaction examplesGoogle Glass

speech recognition

Microsoft HoloLens

gesture & speech recognitionEpson Moverio

head cursor interaction & touch

interaction litmus tests• natural - self explaining, easy to learn • reliable - no false positives, no false negatives • affirmative - appropriate feedback • context sensitive - no misinterpretation • non-intrusive - not bulky, not nerdy

-> ISO 9241

voice interaction

• very flexible • natural • hands free • easy to implement (microphone)

voice interaction

• very flexible • natural • hands free • easy to implement (microphone)

So, what’s the problem?

low reliability!

• flexible • limited natural • sometimes cool

gesture interaction

• flexible • limited natural • sometimes cool

So, what’s the problem?

gesture interaction

learning curve, not hands free, socially awkward,

ambiguous, hard to implement

gesture interaction

but wait - Kinect is super successful, isn’t it?

gesture interactionHaving sold 8 million units in its first 60 days

on the market, Kinect has claimed the Guinness World Record of being the "fastest

selling consumer electronics device".

24 million units of Kinect had been shipped by February 2013.

gesture interactionHaving sold 8 million units in its first 60 days

on the market, Kinect has claimed the Guinness World Record of being the "fastest

selling consumer electronics device".

24 million units of Kinect had been shipped by February 2013.

*Atkinson, Claire (October 17, 2010). "Microsoft gears up for big launch of Kinect controller-free device for Xbox - NYPOST.com". New York Post. Retrieved October 20, 2010.

- „hard“ bundling Kinect with Xbox One failed - compare: in the last 7 years 420 million gaming units were sold - Microsoft had an advertising budget of US$500 million for the launch of Kinect*.

BUT

gesture interaction

Myo Armband

gaining traction

Leap Demo Myo+Sphero

Demo

Gesture Control in

Cars

Leap with Rift

head cursor interaction

universe2go Demo

Webseite Demo

• use well known technology (smartphone) • very reliable • easy to learn (guided UI) • easy to implement

wearable interactionsmartphone

• use well known technology (smartphone) • very reliable • easy to learn (guided UI) • easy to implement

So, what’s the problem?

wearable interaction

then, why use smart glasses at all?

smartphone

• cool? form factor • hands free • can by made quite small and/or fashionable

wearable interactionbracelet

• cool? form factorSo, what’s the problem?

wearable interaction

did you recognize the flaw?

bracelet

• cool? form factor

wearable interactionring

Ring Demo

• cool? form factor

So, what’s the problem?

wearable interaction

still to be explored, learning curve, still bulky

ringRing

Demo

• very reliable • well known • easy to implement

wearable interactionjoystick

• very reliable • well known • easy to implement

So, what’s the problem?

wearable interaction

super nerdy %-D

joystick

• well known device - socially accepted • technology gains market share • reliable • touch easy to implement

wearable interactionsmart watch

• well known device - socially accepted • technology gains market share • reliable • touch easy to implement

So, what’s the problem?

wearable interaction

how?

smart watch

multimodal interaction

voice

gesture

joystick

ring

braceletsmartphone

watch

head/eye cursor

something missing?

CONTEXTsocial point of view

timelocation

mepersonalization

interaction

the fun part

pointingLightsaber

Demo• view direction +

point direction = point of interest

• + very natural (even for two years old) • + reliable • + easy to implement Pebble U2G

Demo

tiltingSpace Demo

• + quite natural for navigational task • + reliable • + easy to implement

proximity

• relative location in reference to a known point • proximity = context • + very natural • - low precision

the perfect triple

direction location

action

„no interaction is an island“

absolute (geo) relative (indoor) proximity

view point move

touch gesture voice

technical challenges

direction location

action

absolute (geo) relative (indoor)proximity

viewpointmove

touchgesture voice

technical challenges

• absolute horizontal direction unreliable (compass) • indoor location unreliable (radio interference) • speech recognition either needs high computing

power or fast connection • touch: how to sense a finger click with least effort;

Myo - too bulky; Ring - too bulky; smart watch - needs second hand (what about an Myo-Pebble Smart Strap? :-)

applications (c)nature / traveling

city / architecturemuseum

smart home

gaming

social

smart glasses user interaction will be very

application specific!

smart home exampletask context action context

marker

switch light pov & point „click“ or speech -

configure device proximity long „click“ -

configure scenario

location, time, device status

double long „click“

favourite: phrase or dedicated

button

activate scenario location, time „double click“

and speechfavourite: phrase

or dedicated button

nature example

task context action

show feature info pov & point -

show trail info proximity „click“

city exampletask context action

show feature info location & pov & point -

show poi location & pov „click"

show directions location head curser input

museum exampletask context action

show feature info location & pov & point -

show poi location & pov „click"

show directions location head curser input

social example

task context action

AR selfie location & pov & point

„click“ and select via head cursor input

gaming example

may the force be with you :-)

to be sure…

• pointing is not limited to pointing to physical objects - you can also point to virtual objects e.g. a virtual keyboard for text input or virtual buttons

• universe2go

• AR: Ori Inbar - Smart Glasses Report 2015; Augmented Reality World Expo AWE 2015 (search Youtube)

• Smart Glasses: Microsoft Hololens, Epson Moverio, Google Glass, Magic Leap, Vuzix, ODG R7, Meta, Cast AR, Seer, Reckon, Optinvent, Atheer, Innovega, Sony SmartEyeglass, Lumus

• Wearables: Myo, Cicret, Ring, Nod Ring

• Smart Watch: Pebble (SDK)

• Gesture Control: Leap Motion, Thalmic Labs - Myo, CNET - Gesture control in cars

• Space Control: Rfduino, Arduino, Eclipse, RXTX Comm Lib (do not use the original one with OS X 10.10), Serial Java Code, Starry Night, Celestia

• Smart Home: Oomi, Nest, Alljoyn

• IMU: Invensense MPU 9150; Razor IMU; Free IMU; Damien Douxchamps List; DCM Theory; Madgewick AHRS

• Misc: Sphero, Don’t click it, ISO 9241, Custom Lightsaber

resources

a brave jump into the wildSmart Glasses will be controlled by

the view direction of the glasses together with a wearable which is a

kind of bracelet, ring or smart watch, which can detect „finger click“, pointing direction and tilting. In conjunction with time, location,

proximity and personalization this will generate an easy and reliable

user experience.

Martin Neumann

innospiring GmbH mobil: +49 160 97710054 martin.neumann@innospiring.de www.innospiring.de Twitter: @innospiring