Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20...
Transcript of Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20...
![Page 1: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/1.jpg)
Chatbot:Controllable Response Generation
Hung-yi Lee 李宏毅
![Page 2: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/2.jpg)
Controllable Response Generation
你要如何貢獻社會?
The same input can have different response.
惡魔貓男:你對社會貢獻的定義…
有為青年:我要好好學習
ChatbotHow are you?
seq2seq
I am good.
Not limited to chatbot (e.g. TTS)
https://youtu.be/qptqIndt64A
![Page 3: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/3.jpg)
Inconsistent [Li, et al., ACL’16]
Chatbot usually chooses short and boring sentences (e.g. “I don’t know”).
![Page 4: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/4.jpg)
Controllable Response Generation
ChatbotHow are you?
seq2seq
?????
Persona Emotion(say something “angry”)
Character
(pretend you are “Trump”)
![Page 5: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/5.jpg)
Controllable Response Generation
Characters[Wang, et al., EMNLP’17]
[Zhou, et al., AAAI’18]
Emotional Chatting Machine
![Page 6: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/6.jpg)
Approach 1: Directly Fine-tune
ChatbotHow are you?
seq2seq
I am good.
• Fine-tuning with limited data is easy to overfit.
• MAML: initial parameters only need a few dialogue samples to adapt. [Lin, et al., ACL’19]
xxx: …..A: ……xxx: …..A: ……
xxx: …..B: ……xxx: …..B: ……
xxx: …..C: ……xxx: …..C: ……
MAML: https://www.youtube.com/watch?v=EkAqYbpCYAc
![Page 7: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/7.jpg)
Approach 2: Control by Condition
How is today
Today is awesome
1
Training
1: positive
Input: How is today?Response: Today is awesome.
(Positive)
Input: How is today?Response: Today is bad.
(Negative)
0: negative
![Page 8: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/8.jpg)
Approach 2: Control by Condition
How is today
Today is bad
0
Training
1: positive
0: negative
Input: How is today?Response: Today is bad.
Input: How is today?Response: Today is awesome.
(Positive) (Negative)
![Page 9: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/9.jpg)
Approach 2: Control by Condition
?
Testing
You decide
? ? ?
I love you
?
Response: I love you, too.
= 0.0
? = 1.0
Response: I am not ready to start a relationship.
![Page 10: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/10.jpg)
Approach 2: Control by Condition
• Persona-based model[Li, et al., ACL’16]
![Page 11: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/11.jpg)
Approach 2: Control by Condition
• Emotional Chatting Machine
[Zhou, et al., AAAI’18]
![Page 12: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/12.jpg)
Approach 2: Control by Condition
• Conditional Transformer Language model (CTRL)[Keskar, et al., arXiv’19]
Prepend the control code in front of token sequence.
w1 w2 w3 ……Bookshttp://speech.ee.ntu.edu.tw/~tlkagk/ w1 w2 w3 ……
![Page 13: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/13.jpg)
![Page 14: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/14.jpg)
![Page 15: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/15.jpg)
Approach 2: Control by Condition
[Zhang, et al., ACL’18]
PERSONA-CHAT
![Page 16: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/16.jpg)
Approach 2: Control by Condition
• TransferTransfo [Wolf, et al., NeurIPS workshop’18]
• Order of personality sentences doesn’t matter
• The dataset can be augmented by different orders
personality input response
![Page 17: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/17.jpg)
Approach 3: Monologues Only
• The previous approaches need the dialogues with specific characteristics.
• What if we only have monologues
![Page 18: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/18.jpg)
Approach 3: Monologues Only
• Rank: selecting sentences from monologue as response
• Multiply:
• Train a language model by monologue
• Using the language model to influence chatbot response
• Pseudo Data:
[Wang, et al., EMNLP’17]
Monologue
I want to build a wall.
Make America great againChatbotinput response
Reverse inputresponseWhat do you want to do?
What’s your goal in life?
→ Dialogue
![Page 19: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/19.jpg)
Approach 3: Monologues Only
[Luan, et al., IJCNLP’17]
Monologue
![Page 20: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/20.jpg)
Approach 3: Monologues Only
How is today
Today is bad
Classifier
Update parameter to fool the classifier
monologue
In monologue or not?
Extra Constraint • Train a model to predict input from response
• Constrain the amount of update
(by RL or GAN)
![Page 21: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/21.jpg)
Approach 3: Monologues Only
How is today
Today is bad
SentimentClassifier
monologue
positive sentences
0.1
good
Update parameter to fool the classifier
Extra Constraint • Train a model to predict input from response
• Constrain the amount of update
(by RL or GAN)
![Page 22: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/22.jpg)
Positive sentencesDemo
• Human: yeah, i’m just less interested in the how-to and more in the what and why of the things we build
• General Chat-bot: it’s not a good thing .
• Positive Chat-bot : it’s a good one
• Human: always learning and self improving is a huge turn on for me so believe me i understand
• General Chat-bot: i'm not sure if i can see you in a relationship
• Positive Chat-bot : i love you so much
[Lee, et al., ICASSP, 2018]
![Page 23: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/23.jpg)
More examples: https://adelaidehsu.github.io/Personalized-Dialogue-Response-Generation-learned-from-Monologues-demo/
Do you like him?
No, I’m sorry.
He’s a good kid.No.
He’s a really good friend.
He’s a cool kid.
He’s my man.
No, just stop. General Response
Demo [Su, et al., INTERSPEECH’19]
![Page 24: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/24.jpg)
[Dathathri, et al., ICLR’20]
Plug & Play Language Models
![Page 25: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/25.jpg)
[Dathathri, et al., ICLR’20]
![Page 26: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/26.jpg)
Next Step
Control the response of chatbot
Control the response of interlocutor
(positive)
[Shin, et al., ICASSP’20]
(positive)
e.g. always say something positive
vv
v
![Page 27: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/27.jpg)
Concluding Remarks
Approach 1: Directly Fine-tune
Approach 2: Control by Condition
Approach 3: Monologue only
![Page 28: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/28.jpg)
Reference
• Yi Luan, Chris Brockett, Bill Dolan, Jianfeng Gao, Michel Galley, Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models, IJCNLP, 2017
• Thomas Wolf, Victor Sanh, Julien Chaumond, Clement Delangue, TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents, NeurIPS Workshop, 2018
• Di Wang, Nebojsa Jojic, Chris Brockett, Eric Nyberg, Steering Output Style and Topic in Neural Response Generation, EMNLP, 2017
• Zhaojiang Lin, Andrea Madotto, Chien-Sheng Wu, Pascale Fung, Personalizing Dialogue Agents via Meta-Learning, ACL, 2019
• Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, Richard Socher, CTRL: A Conditional Transformer Language Model for Controllable Generation, arXiv, 2019
• Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, Jason Weston, Personalizing Dialogue Agents: I have a dog, do you have pets too?, ACL, 2018
• Feng-Guang Su, Aliyah Hsu, Yi-Lin Tuan and Hung-yi Lee, "Personalized Dialogue Response Generation Learned from Monologues", INTERSPEECH, 2019
![Page 29: Chatbot: Controllable Response Generationspeech.ee.ntu.edu.tw › ~tlkagk › courses › DLHLP20 › Chatbot (v6).pdf · Demo Positive sentences •Human: yeah, i’mjust less interested](https://reader036.fdocuments.net/reader036/viewer/2022062402/5f27bbfa51d03418e20bc9ee/html5/thumbnails/29.jpg)
Reference
• Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, Rosanne Liu, Plug and Play Language Models: A Simple Approach to Controlled Text Generation, ICLR, 2020
• Jamin Shin, Peng Xu, Andrea Madotto, Pascale Fung, Generating Empathetic Responses by Looking Ahead the User’s Sentiment, ICASSP, 2020
• Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, Bing Liu, Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory, AAAI, 2018
• Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, Bill Dolan, A Persona-Based Neural Conversation Model, ACL, 2016