Multimodal Interactions & JS: The What, The Why and The How - Diego Paez, Despegar
-
Upload
nodejsfoundation -
Category
Technology
-
view
39 -
download
2
Transcript of Multimodal Interactions & JS: The What, The Why and The How - Diego Paez, Despegar
Multimodal InteractionsThe what, the why and the how
Diego Paez, Despegar
About Me
Diego Paez => DEKΔ
G E U Tco-founder
JS developer with a passion for HCI ♥�
Currently working at:
AGENDA
➔ Definitions
➔ Foundations
➔ An alternative approach
➔ Future work
What is a multimodal interaction (MMI)?
#The what
Human ComputerInteraction
Feedback&
Senses
5 major senses: sight, hearing, touch, taste and smell. But there are more; eg: Proprioception.
You Real World
Interaction
Highway 401 by Nayuki
Senses
But, why?
#The why
Why?
➔ Lowering barriers to HCI. Eg: gestures research.
➔ Accessibility [1]➔ Killer feature➔ IoT
Foundations
#The how
Architecture proposed
by Dumas et al [2]
1
2
3
An approach
#The how
Plusultra Gyes
Proposed Architecture
Modalities(HW, distributed)
Pub/sub pattern used to separate
layers
Web Clients (distributed)
1
2
3
Proposed Architecture
Getting started
npm install plusultraDocker-compose up
mkdir myMultimodalDeviceDrivercd myMultimodalDeviceDrivernpm install gyes
Creating a MM Device Drivervar Inherits = require( 'inherits' );var ModalityChannel = require( 'gyes' ).ModalityDriver;
module.exports = MyModality;
function MyModality( opts ){// constructor code...
}
Inherits(MyModality, ModalityChannel);
MyModality.prototype.fission = function( time ){ … };
MyModality.prototype.newEvent = function( data ){ … };
Inside the app// lets create a new plusultra clientplusultraClient = new gyes( app_key, socketURI, options );plusultraClient.authenticate( app_key );
// create a new modality (this is like metadata)var myMod = new gyes.Modality( modName, 'input', {} );
// instance a driver for the modalityvar modDriver = new myMultimodalDeviceDriver( options );myMod.use( modDriver );
// finally add the modality into the systemplusultraClient.addModality( app_key, airMod );
Future Workaka: Help is welcome!
➔ Experimenting with WebRTC to enable P2P interactions.
➔ Define Interpretations as components. W3C MMI. React can
help?
➔ Improve API. Follow a reactive/functional approach. Cycle/rxjs?
Future Work
Project repos➔ Plusultra: https://github.com/dpaez/plusultra
➔ Gyes: https://github.com/dpaez/gyes
➔ Demo app (shapes): https://github.com/dpaez/shapes_app
➔ Gitter: https://gitter.im/gyes_mmi/Lobby
Thanks!Who is the hipster?Follow me @caraxGithub @dpaezGet the GEUT experience: geutstudio.com / Some cool code @geut