Alternative smartphone interactions

27
@michaelpoltorak Michael Poltorak Nielsen @michaelpoltorak Alternative Smartphone Interactions Based on Sensor Data

description

Slides from my talk about alternative smartphone interactions at jsconf.eu 2014

Transcript of Alternative smartphone interactions

Page 1: Alternative smartphone interactions

@michaelpoltorak

Michael Poltorak Nielsen@michaelpoltorak

Alternative SmartphoneInteractions Based on Sensor Data

Page 2: Alternative smartphone interactions

@michaelpoltorak

Page 3: Alternative smartphone interactions

@michaelpoltorak

I work at Statsbiblioteket

Page 4: Alternative smartphone interactions

@michaelpoltorak

What did he say?

Page 5: Alternative smartphone interactions

@michaelpoltorak

It’s a library

Page 6: Alternative smartphone interactions

@michaelpoltorak

Statsbiblioteket, in Aarhus, Denmark

University research library and national libraryHuge search index (solr, 1 BN records)Keeps a physical copy of all Danish stuff (books, newspapers, flyers, etc.) for cultural, preservational purposesHarvests .dk domain for Danish web archiveStores Danish radio and TV (daily 1 TB, so far 1.5 PB)Newspaper digitisation: 32 M pages, 600 TB

Page 7: Alternative smartphone interactions

@michaelpoltorak

Loading more slides

DONE

Page 8: Alternative smartphone interactions

@michaelpoltorak

Mobile javascript sensor API’s and events

Page 9: Alternative smartphone interactions

@michaelpoltorak

Page 10: Alternative smartphone interactions

@michaelpoltorak

Touch screen. Touch events All browsers

Location. Navigator.geolocation All browsers

Motion. DeviceOrientation, DeviceMotion All browsers

getUserMedia/StreamAPI FF, Ch for Android, Bb, OM

Speech. Web Speech API Ch for Android, iOS < 7

Proximity (UserProximityEvent, DeviceProximityEvent) FF

Ambient Light API (device's light sensor) FF

Battery Status API FF

Page 11: Alternative smartphone interactions

@michaelpoltorak

Introducing the

Demo search engine

Page 12: Alternative smartphone interactions

@michaelpoltorak

A closer look at:touch,

device motion,web speech API

Page 13: Alternative smartphone interactions

@michaelpoltorak

TouchThree types:

touchstart, touchend, touchmoveno of fingers, position on screen, drag(how long)

IE: pointer events (mouse + pen + touch)

Page 14: Alternative smartphone interactions

@michaelpoltorak

TouchEvent {type: "touchmove", //fired when draggingtarget: DOM element

touches: TouchList [// size: no fingers{

clientX: 97, //position on screenclientY: 259,radiusX: 11.5, //finger shaperadiusY: 11.5

}]}

Page 15: Alternative smartphone interactions

@michaelpoltorak

1

$1 Touch gesture recognizer

Page 16: Alternative smartphone interactions

@michaelpoltorak

var result = ui.recogniser.Recognize(touchPoints, false); //do touchPoints resemble a $1-gesture?

if (result.Score > 0.8) { touchName = (result.Name); touch.command(result.Name);

}

Page 17: Alternative smartphone interactions

@michaelpoltorak

Device orientation event:

window.addEventListener('deviceorientation', function(eventData) {

eventData.gamma; //left-to-right tilteventData.beta; //front-to-back tilteventData.alpha; //compass direction

}, false);

IE10: Windows 8 Sensor API

Page 19: Alternative smartphone interactions

@michaelpoltorak

3d gestures from device orientation event data:

if (delta_gamma > 30) { //positive gamma changereturn "TILT_RIGHT";

}if (delta_gamma < -30) {//negative gamma change

return "TILT_LEFT";}

Page 20: Alternative smartphone interactions

@michaelpoltorak

Web Speech API //Android Chrome only

var recognizer = new webkitSpeechRecognition(); recognizer.onresult = function (data) {

var words,list; list=data.results.SpeechRecognitionResultList;words =[list.length - 1][0].transcript;

console.log("Heard words ", words); };

Page 21: Alternative smartphone interactions

@michaelpoltorak

Search engine demoNow with:

custom touch gestures3d gestures

speech

Page 22: Alternative smartphone interactions

@michaelpoltorak

So, we can augment the traditional paradigm

Page 23: Alternative smartphone interactions

@michaelpoltorak

But is it useful?

Page 24: Alternative smartphone interactions

@michaelpoltorak

Touch gestures: difficult on small/scrolling screen, no common set of gestures. 3D gestures: more intuitive, but hard to do. No common language, hard to code (noise, hard to keep apart).Speech: easy to speak, but hard to decode. Mistakes will occur. Disturbing other people. You look foolish speaking to your device.

Page 25: Alternative smartphone interactions

@michaelpoltorak

Very important:Evaluate any new features,

test on users,and, as always:

Page 26: Alternative smartphone interactions

@michaelpoltorak

Kill your darlings

Page 27: Alternative smartphone interactions

@michaelpoltorak

Thank you!

@michaelpoltorakmichaelpoltorak.dk