An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User...
Transcript of An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User...
![Page 1: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/1.jpg)
An Approach for Ex-Post-Facto Analysis of KnowledgeGraph-Driven Chatbots – the DBpedia Chatbot
Rricha Jalota, Priyansh Trivedi, Gaurav Maheshwari, Axel-Cyrille Ngonga Ngomo, RicardoUsbeck
November 20, 2019
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 1 / 17
![Page 2: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/2.jpg)
Introduction
Figure: U.S. Chatbot Market by Vertical, 2014 - 2025 (USD Million) 1
1Source: https://www.grandviewresearch.com/industry-analysis/chatbot-marketJalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 2 / 17
![Page 3: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/3.jpg)
IntroductionKnowledge Graphs and Knowledge Graph-Driven Systems
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 3 / 17
![Page 4: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/4.jpg)
BackgroundKnowledge Graph-Driven Chatbot: The DBpedia Chatbot
Deployed in August 2017
Purpose2
- Answer factual questions- Answer questions related to DBpedia- Expose the research work being done in DBpedia as product features- Casual conversation/banter
Hybrid Chatbot - domain-specific information (DBpedia-centric FAQs) +domain-agnostic factual questions (using DBpedia KG)
2Source: https://wiki.dbpedia.org/blog/meet-dbpedia-chatbotJalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 4 / 17
![Page 5: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/5.jpg)
BackgroundKnowledge Graph-Driven Chatbot: The DBpedia Chatbot
Deployed in August 2017
Purpose2
- Answer factual questions- Answer questions related to DBpedia- Expose the research work being done in DBpedia as product features- Casual conversation/banter
Hybrid Chatbot - domain-specific information (DBpedia-centric FAQs) +domain-agnostic factual questions (using DBpedia KG)
2Source: https://wiki.dbpedia.org/blog/meet-dbpedia-chatbotJalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 4 / 17
![Page 6: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/6.jpg)
BackgroundCase Study: The DBpedia Chatbot
Total: 9084 users, 90,800 interactions
Table: Feedback Statistics
Feedback-asked 28953Feedback-received 7561Negative-feedback 4155
Figure: Architecture of the DBpedia Chatbot
Check http://chat.dbpedia.orghttps://github.com/dbpedia/chatbot
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 5 / 17
![Page 7: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/7.jpg)
Objective of the Ex-Post-Facto Analysis
Understand the nature of user-requests
- query-patterns- user-intentions
Examine whether the chatbot can serve its purpose – satisfy user-requests
Get insights about the conversation flow to improve the chatbot’s architecture
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 6 / 17
![Page 8: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/8.jpg)
Approach Overview
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 7 / 17
![Page 9: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/9.jpg)
ApproachRequest Analysis - Intent Analysis
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 8 / 17
![Page 10: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/10.jpg)
ApproachRequest Analysis - Intent Analysis
Figure: Visualization of clusters obtained via HDBSCAN on sentence embeddings. Each cluster consistsof at least 25 samples. The top 10 clusters out of a total of 33 have been labeled with their top terms.
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 9 / 17
![Page 11: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/11.jpg)
ApproachRequest Analysis - Complexity of utterances
Complex Query
Example: Can you give me the names of women born in the Country during the 19th century?
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 10 / 17
![Page 12: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/12.jpg)
ApproachRequest Analysis - Miscellaneous Analysis
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 11 / 17
![Page 13: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/13.jpg)
Response Analysis
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 12 / 17
![Page 14: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/14.jpg)
Response AnalysisEntity Types in Utterances prior to Negative Feedback
Figure: Entity type distribution from 1000manually annotated failed utterances.
Table: spaCy-NER and DBpedia Spotlightaccuracy for detecting person and locationmentions.
System Person Location
spaCy-NER 41.3% 42.2%DBpedia Spotlight 69.2% 46.1%
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 13 / 17
![Page 15: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/15.jpg)
Conversation Analysis
Figure: Topics as identified by DBpedia Spotlight
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 14 / 17
![Page 16: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/16.jpg)
Implications for DBpedia Chatbot
Adding support for multilingualism
Smart Suggestions
Detecting implicit feedback andout-of-scope queries
Knowledge-based QA
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 15 / 17
![Page 17: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/17.jpg)
Implications for Knowledge-driven Chatbots
Multilingual Support
Guide User Input
Guiding User Expectations
Adding explainability
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 16 / 17
![Page 18: An Approach for Ex-Post-Facto Analysis of Knowledge Graph ......Multilingual Support Guide User Input Guiding User Expectations Adding explainability Jalota et al Analysis of Knowledge](https://reader035.fdocuments.net/reader035/viewer/2022071115/5ffb5b50b52333447030abec/html5/thumbnails/18.jpg)
That’s all Folks!
Get in touch:
Rricha JalotaData Science Group, Paderborn University
[email protected]/dice-group/DBpedia-Chatlog-Analysis
Follow us on Twitter: @DiceUPB, @FraunhoferIAIS,@RrichaJalota
Jalota et al Analysis of Knowledge Graph-Driven Chatbots November 20, 2019 17 / 17