Introduction to Talking Robots - ling.helsinki.figwilcock/Doshisha-2016/docs/5-SpeechDialogu… ·...
Transcript of Introduction to Talking Robots - ling.helsinki.figwilcock/Doshisha-2016/docs/5-SpeechDialogu… ·...
www.helsinki.fi/yliopisto
Introduction toTalking Robots
Graham Wilcock
Adjunct Professor, Docent Emeritus
University of Helsinki
2/12/2016Graham Wilcock / Introduction to Talking Robots 1
www.helsinki.fi/yliopisto
Robot Speech and Dialogues
Graham Wilcock
2/12/2016Graham Wilcock / Introduction to Talking Robots 2
www.helsinki.fi/yliopisto
Speech Recognition:Open or Closed Vocabulary?
• Closed vocabulary
• Example: city names (c. 50 flight destinations)
• [ Helsinki | London | Paris | Berlin | Bern | Dublin | … ]
• How many words in a language?
• English: estimated 300,000 different words
• Open vocabulary
• Example: words that sound like ”house”
• [ grouse | browse | blouse | ploughs | louse | … ]
2/12/2016 3Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
Spoken Dialogue Systems:Closed-Domain Dialogues
• Example: Flight reservations
• Using finite state transitions
• Easily implemented by finite state machine (FSM)
• Fixed-domain database
• Flights, cities, days
• Easy to add new flights, new cities, …
• Difficult to switch domains
2/12/2016 4Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
FSM for a Closed Domain
2/12/2016 5Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
Spoken Dialogue Systems:Annual Conferences
• SIGDIAL (since 2000)
• Special Interest Group on Discourse and Dialogue
• http://sigdial.org/sigdial-meetings
• IWSDS (since 2009)
• International Workshop on Spoken Dialogue Systems
• http://www.iwsds.org/
2/12/2016 6Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
Spoken Dialogue Systems:Books
• K. Jokinen and M. McTear
Spoken Dialogue Systems
Morgan & Claypool, 2009
• K. Jokinen
Constructive Dialogue Modelling
Wiley, 2009
2/12/2016 7Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
Robot Dialogue Systems:Books
• K. Jokinen and G. Wilcock
(editors)
Dialogues with Social Robots
Springer, 2017
2/12/2016 8Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
WikiTalk: Talking Robots and Wikipedia
2/12/2016 9Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
• Visits to MTA-SZTAKI, Budapest 2011
• Hungarian Academy of Sciences
• Advanced 3-D Virtual Reality with Nao robot
• “It would be nice if it could talk about Shakespeare”
• How could that be possible?
• By using Wikipedia (key idea)
• I made a prototype with Python in 2011
• Windows speech (Windows XP)
2/12/2016
History of WikiTalk (1):Hungary 2011
Graham Wilcock / Introduction to Talking Robots 10
www.helsinki.fi/yliopisto
New Direction:Open-Domain Dialogues
• Need open-domain knowledge source
• Get information from the web
• Wikipedia has articles on almost any topic
• Example: to talk about Shakespeare
• Get Wikipedia article about Shakespeare
• Start reading out the first paragraph
2/12/2016 11Graham Wilcock / Introduction to Talking Robots
2/12/2016 12Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
FSM for Open-Domain (draft)
2/12/2016 13Graham Wilcock / Introduction to Talking Robots
2/12/2016 14Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
Topic Shifts and Topic Chains
• Hyperlinks -> topic shifts -> topic chains
• Robot: ”Shakespeare was born and raised in Stratford-upon-Avon” (link)
• Human: ”Stratford-upon-Avon?” (topic shift)
• Robot: ”Stratford-upon-Avon is a market townand civil parish in Warwickshire, England”
• Human: ”Warwickshire?” (topic shift)
• Otherwise, continue same topic
2/12/2016 15Graham Wilcock / Introduction to Talking Robots
2/12/2016Graham Wilcock / Introduction to Talking Robots 16
www.helsinki.fi/yliopisto
FSM for Open-Domain (WikiTalk)
2/12/2016 17Graham Wilcock / Introduction to Talking Robots
2/12/2016 18Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
Speech Recognition in WikiTalk
• Open Domain but not Open Vocabulary
• Open vocabulary speech recognition basically does not work (about 300,000 words in English)
• Recognize phrases from small phrase sets
• Dynamically predicted from Wikipedia links
• Recognize small set of navigation commands
• ”Repeat”, ”Continue”, ”Enough”, ”Previous”, etc.
2/12/2016 19Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
• International summer school project, Metz 2012
• Led by Prof. Kristiina Jokinen (Tartu U.) and me
• 5 PhD students, 4 Nao robots, 4 hot weeks in July
• Transferred Python WikiTalk code to Nao robots
• Added robot gesturing and multimodal interaction
• Demos in Metz and Budapest 2012
2/12/2016
History of WikiTalk (2):France 2012
Graham Wilcock / Introduction to Talking Robots 20
www.helsinki.fi/yliopisto 2/12/2016
International Summer Workshop eNTERFACE 2012, Metz, France
Graham Wilcock / Introduction to Talking Robots 21
www.helsinki.fi/yliopisto 2/12/2016
Making WikiTalk on 4 Nao robots at eNTERFACE 2012
Graham Wilcock / Introduction to Talking Robots 22
www.helsinki.fi/yliopisto 2/12/2016
First Demo of WikiTalk on Nao (Metz, France 2012)
Graham Wilcock / Introduction to Talking Robots 23
https://www.youtube.com/watch?v=A7XFcnJE29A
www.helsinki.fi/yliopisto 2/12/2016 24Graham Wilcock / Introduction to Talking Robots
ATIS Spelling Alphabet(used with ”Alphabet” command)
www.helsinki.fi/yliopisto
• Finnish WikiTalk, Helsinki 2014
• Prof. Jokinen, PhD student Niklas Laxström and me
• 2 robots borrowed from SASKY, Sastamala
• Demo at Robottiviikko 2014 exhibition
• Japanese WikiTalk, Kyoto 2015
• Prof. Yamamoto, MSc student Okonogi and me
• 3 robots in Prof. Yamamoto’s lab, Doshisha University
• Demo at SIGDIAL 2015 conference
2/12/2016
History of WikiTalk (3):Multilingual versions
Graham Wilcock / Introduction to Talking Robots 25
www.helsinki.fi/yliopisto
EU Robotics Week 2014at ”AiRo Island” Helsinki
2/12/2016 26Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
Multilingual WikiTalk:Finnish and English
2/12/2016 27
https://www.youtube.com/watch?v=q0aSgPns4dE
Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
Multilingual WikiTalk:English and Japanese
2/12/2016 28
https://www.youtube.com/watch?v=NkMkImATfYQ
Graham Wilcock / Introduction to Talking Robots
www.helsinki.fi/yliopisto
• Talking Robot + Wikipedia
• Instant talking on any topic (open-domain)
• Community-validated talking
• Extend robot communicative capabilities
2/12/2016 29Graham Wilcock / Introduction to Talking Robots
Open-Domain Talking:WikiTalk
www.helsinki.fi/yliopisto
• Anticipated developments in ASR
• Continuous listening capability
• Better cloud-based speech recognition
• Open vocabulary speech recognition
• Wikipedia-based listening
• Open-domain named entity recognition
• Wikification: Wikipedia reference linking
• Extend robot communicative capabilities again
2/12/2016 30Graham Wilcock / Introduction to Talking Robots
Open-Domain Listening:Towards WikiListen
www.helsinki.fi/yliopisto
• Example: North Sami
• Spoken by 20,000 people in Lapland
• DigiSami project
• To support revitalization of North Sami language usingmodern language technologies
• Prof Kristiina Jokinen (University of Helsinki)
• Funded by Academy of Finland
2/12/2016 31Graham Wilcock / Introduction to Talking Robots
Speech Technology for Under-resourced Languages
www.helsinki.fi/yliopisto 2/12/2016 32Graham Wilcock / Introduction to Talking Robots
The Sami Languages
www.helsinki.fi/yliopisto 2/12/2016 33Graham Wilcock / Introduction to Talking Robots
DigiSami Corpus of North Sami Conversational Speech
www.helsinki.fi/yliopisto 2/12/2016 34Graham Wilcock / Introduction to Talking Robots
Speech Synthesizer and Recognizer for North Sami
https://www.youtube.com/watch?v=14jYeViJ0X0
www.helsinki.fi/yliopisto 2/12/2016 35Graham Wilcock / Introduction to Talking Robots
First Demo of SamiTalk(IWSDS 2016 in Lapland)
https://drive.google.com/file/d/0B-D1kVqPMlKdeXhXbnlUMmtYcU0