Embedding Intelligence in Everyday Objects with TJBot
-
Upload
victor-dibia -
Category
Devices & Hardware
-
view
411 -
download
1
Transcript of Embedding Intelligence in Everyday Objects with TJBot
IBM Research
1
IBM Research
Victor Dibia
Embedding Intelligence in Everyday Objects with TJBot.An open source DIY project powered by Watson Cognitive Services.
Human – Agent Collaboration Lab, IBM [email protected]@vykthur | github.com/victordibia
Feb 20, 2017
IBM Research
2
TJBot : What and Why?- Open source DIY project to get you engaged
with Watson Services
IBM Research
3What is TJBot?- A cardboard robot
- Simple, approachable- Open Source (design, code)- Cognitive (IBM Watson services)- Extensible (prototyping platform)
Components: Raspberry Pi, LED, Camera, Microphone, Speaker, Servo.
IBM Research
4
3D Print Laser cut
IBM Research
5ibm.biz/mytjbot
IBM Research
6Recipes.Step by step instructions +Code (node.js) to help you prototype capabilities for TJBot powered by Watson services.
http://www.instructables.com/member/TJBot/
IBM Research
7Project GoalsHow can we make it easier to engage a community of enthusiasts experimenting with embodied cognition – the idea of embedding intelligence in everyday objects within the physical world?
IBM Research
8Project GoalsDesign principle – Approachable Design
- Use of familiar material (cardboard) that can be altered with ease.
- Simplified part assembly: no soldering or adhesive required.
- Simplified programming model and language interface (JavaScript).
IBM Research
9Project OutcomeA prototyping platform to help democratize Embodied Cognition.
Target communities:
- Makers- Developers- Students (Education and Learning)
IBM Research
10How Does Watson Enable TJBot?ListenWatson Speech to Text service converts spoken speech to text that can be analyzed
SpeakWatson Text to Speech service service converts text to sound using various voices.
Understand EmotionsWatson Tone Analyzer service can infer the emotion within text. E.g.. it can tell if a message contains emotions like happy , sad, angry
Understand ConversationsWatson Conversation Service can respond to users in a way that simulates a conversation between humans.
SeeWatson Visual Recognition service can understand the content of an image and describe it.
IBM Research
11TJBotSensors
ExampleCapabilities
ExampleWatson Services
ExampleUse cases
LED
Speakers
Camera Servo Motor Arm
Microphone Listen
Speak
Shine
Show emotion
Wave
See
Speech to text
Tone Analyzer
Vision Recognition
Conversation
Text to speech
Sentiment Analysis
Virtual Agents (eldercare, home care)
Education (language learning)
IBM Research
12
Demo.- Watson Services
- Speech to text- Text to speech- Conversation- Visual Recognition
IBM Research
13
Overview of Watson Services
IBM Research
14
IBM Watson Cognitive
Take your first step into the cognitive era with our variety of smart services.
Services.
- Natural interaction- Semi-structured data processing- Trained and continuously improved via machine learning and deep
learning.- Restful API services with SDKs for node.js, java, python.
IBM Research
15
Language
IBM Research
16
Speech
Vision
Data Insights
IBM Research
17
Speech
Vision
Data Insights
IBM Research
18Speech to TextConverts audio voice into written text.
• Transcription • Voice-controlled applications: allows for custom
models
https://speech-to-text-demo.mybluemix.net/
IBM Research
19Text To SpeechConverts written text into natural sounding audio in a variety of languages and voices. • Customize and control the pronunciation of specific words to
deliver a seamless voice interaction that catered s to your audience.
• Interactive voice based applications.
https://text-to-speech-demo.mybluemix.net/
IBM Research
20Tone AnalyzerUses linguistic analysis to detect three types of tones in written text: emotions, social tendencies, and writing style. • Understand emotional context in conversations or
communications• Taylor interaction based on sentiment.
https://tone-analyzer-demo.mybluemix.net/
IBM Research
21Visual RecognitionUnderstands the contents of images - visual concepts tag the image, find human faces, approximate age and gender, and find similar images in a collection. • Train the service by creating your own custom concepts.
Use Visual Recognition to detect a dress type in retail, identify spoiled fruit in inventory, and more.
https://visual-recognition-demo.mybluemix.net/
IBM Research
22AlchemyLanguage
Analyzes text to help you understand its concepts, entities, keywords, sentiment, and more.
• Additionally, you can create a custom model for some APIs to get specific results that are tailored to your domain.
https://alchemy-language-demo.mybluemix.net/
IBM Research
23ConversationQuickly build, test and deploy a bot or virtual agent across mobile devices, messaging platforms like Slack or even on a physical robot.
• Visual dialog builder to help you create natural conversations between your apps and users, without any coding experience required.
https://conversation-demo.mybluemix.net/
IBM Research
24
Programming TJBot- Getting started - Tying stuff together
IBM Research
25Steps.
- Wifi Setup, - Raspberry Pi Software update - Hardware setup : LED, Servo, Microphone, etc- Credential Setup : Bluemix- Recipe software setup : Clone Github repo - Ready to go.
IBM Research
26
http://www.instructables.com/member/TJBot/Instructions
IBM Research
27Libraries UsedDepends on several npm packages.
- RGB LED – ws281x library- Servo – pigpio software PWM library- Microphone – mic library- Speaker – aplay library- Camera – raspistill wrapper
IBM Research
28Code Walk through: Control LED on TJBot using voice.- http://www.instructables.com/i
d/Use-Your-Voice-to-Control-a-Light-With-Watson/
- Code Walk through
IBM Research
29
TJBot Library [Beta]- Experimental work to encapsulate basic
functions of the bot.- https://github.com/ibmtjbot/tjbotlib
IBM Research
30The TJBot Library
Encapsulate basic functions for TJBot such as listening, speaking, led color change, waving, seeing.
IBM Research
31The TJBot Library
tj.listen(transcript callback)tj.speak(“text”)tj.converse()tj.see()tj.shine(“red”)
IBM Research
32Code Walk through: Control LED using the TJBot library.
- https://github.com/ibmtjbot/recipes
- Code Walk through
IBM Research
33
Open Issues- Improving accuracy- Bot “Interruptibility” - Gracefully managing latency
IBM Research
34Improving AccuracyHow do we improve interaction (voice) accuracy? Improving Speech-to-Text models may not be enough!
- Customized language models?- Intent Matching?- Multi-turn conversations?
IBM Research
35Bot “Interruptibility”When and how should the robot be interrupted (while performing an activity like speaking, waving etc.)?
- Vision? (monitoring a user’s facial expression, raised hand)
- Hardware button or sensor?
IBM Research
36Latency ToleranceLatency can severely degrade quality of interaction. How do we minimize its effect?
- Managing and ordering service responses- Leverage cues to provide additional information- Balancing capabilities – cloud vs local
processing.
IBM Research
37
Next Steps
IBM Research
38Next Steps3 pronged
- Conduct basic research that address open issues.
- Make TJBot simpler and easier to use (tjbotlibrary, visual programming tool)
- Build and sustain the TJBot community.
IBM Research
39Learn more?
- Ibm.biz/mytjbot- http://www.instructables.com/
member/TJBot/
IBM Research
40
Thank You!