Mic, Camera, Action!

Shwetank Dixit

Opera Software

about meWeb Evangelist, Web

Standards Team

Member, W3C Mobile Web

for Social Development

Group

Twitter: @shwetank

Email: shwetankd@opera.com

The web is awesome!

but Native apps sometimes have an edge

But that edge is getting smaller

We’re adding more ways to get input from the

user

We’re making existing ways easier

Like web forms

We want to

StopWriting boring JS

We want to

StartWriting exciting JS

Mic

MicDEMO

MicWeb Audio API + getUserMedia

getUserMediaif (navigator.getUserMedia){ navigator.getUserMedia({audio: true}, a_success, a_error);}

getUserMediaMic returns raw audio samplesCannot be played natively by the audio playerNeed Web Audio API to process it and then play it

getUserMediaCreate audio context and media stream source

getUserMedia//Create audio context and media stream source

var audioContext = new AudioContext(); // Prefix it for various browsers

// Create an AudioNode from the stream.var mediaStreamSource = audioContext.createMediaStreamSource( stream );

getUserMedia//Create audio context and media stream source

var audioContext = new AudioContext(); // Prefix it for various browsers

// Create an AudioNode from the stream.var mediaStreamSource = audioContext.createMediaStreamSource( stream );

// Connect audio loop to hear your own voicemediaStreamSource.connect(context.destination);

Recorder.jsGreat library to help you record mic

input using getusermedia

Recorder.jsbuffers the raw audio samples.

converts it to format which can be

played by an <audio> control

Camera

Use camera with getusermediavar options = {audio: false, video: true};

Use camera with getusermediavar options = {audio: false, video: true};

navigator.getUserMedia(options, v_success, v_error);

Use camera with getusermedia// In the HTMLvar video = document.querySelector('video');.........// In the JS Codefunction v_success(stream){ video.src = window.URL.createObjectURL(stream) || stream;}

Use camera + Video + Canvas for new tricksvar button = document.querySelector('#button'); button.addEventListener('click',snapshot, false);......function snapshot(){ var c = document.querySelector('canvas'); var ctx = c.getContext('2d'); var cw = c.clientWidth; var ch = c.clientHeight; ctx.drawImage(video_element, 0, 0, cw, ch); }

DemoNotice you can click on a polaroid

and save that image. You can create data URIs of canvases

Manipulating the image from the webcam

Once you get the snapshot of the webcam

into canvasyou can go crazy with it

DEMOWarholiser

Action!

Tracking user actionsMotion detection

gesture recognition

headtrackr.jshttps://github.com/auduno/headtrackr/

headtrackr.jshttp://auduno.tumblr.com/post/25125149521/head-tracking-with-

webrtc

Face Detection

X, Y position of face

Width, Height of face relative to canvas

headtrackr.js

Demo

Voice driven apps

Voice modulation apps

Native photo capture in web apps

Live image manipulation and effects

Video conferencing (using WebRTC)Accessibility enhancements to apps

Possible use cases

Cheers!More questions? Ask me now

or contact me at:shwetankd@opera.com

or on twitter - @shwetank