Question Answering (and Textual...
Transcript of Question Answering (and Textual...
QuestionAnswering(andTextualEntailment)
Prof.SameerSinghCS295:STATISTICALNLP
WINTER2017
March14,2017
BasedonslidesfromDanJurafsky,YejinChoi,StephenClark,DanKlein,Niranjan Balasubramanian,andeveryoneelsetheycopiedfrom.
Upcoming…
• Homework4wasdueonlastnight• Lowestgradeofthehomeworks willbedroppedHomework
• Finalreportdueinaweek:March20,2017• Instructionscomingsoon:ACLstyle,5pages(+references)Project
• Papersummariesduetonight• Summary2gradedSummaries
CS295:STATISTICALNLP(WINTER2017) 2
TA/InstructorEvaluationsareavailable!
Outline
QuestionAnswering
TextualEntailment
CS295:STATISTICALNLP(WINTER2017) 3
IR-BasedQASystem
OtherExtensions
Outline
QuestionAnswering
TextualEntailment
CS295:STATISTICALNLP(WINTER2017) 4
IR-BasedQASystem
OtherExtensions
QuestionsinModernSystemsFactoidquestions◦ Whowrote“TheUniversalDeclarationofHumanRights”?◦ Howmanycaloriesarethereintwoslicesofapplepie?◦ Whatistheaverageageoftheonsetofautism?◦ WhereisAppleComputerbased?
Complex(narrative)questions:◦ Inchildrenwithanacutefebrileillness,whatistheefficacyofacetaminopheninreducingfever?
◦ WhatdoscholarsthinkaboutJefferson’spositionondealingwithpirates?
CS295:STATISTICALNLP(WINTER2017) 5
Commercialsystems:mainlyfactoidquestions
WhereistheLouvreMuseumlocated? InParis,France
What’stheabbreviation forlimitedpartnership? L.P.
What arethenamesofOdin’sravens? Huginn andMuninn
What currencyisusedinChina? Theyuan
Whatkindofnutsareusedinmarzipan? almonds
WhatinstrumentdoesMaxRoachplay? drums
WhatisthetelephonenumberforStanfordUniversity? 650-723-2300
CS295:STATISTICALNLP(WINTER2017) 6
ParadigmsforQA
IR-basedapproaches◦ TREC;IBMWatson;Google
Knowledge-basedandHybridapproaches◦ IBMWatson;AppleSiri;WolframAlpha;TrueKnowledgeEvi
CS295:STATISTICALNLP(WINTER2017) 7
Manyquestionscanalreadybeansweredbywebsearch
CS295:STATISTICALNLP(WINTER2017) 8
IR-basedQuestionAnswering
CS295:STATISTICALNLP(WINTER2017) 9
IR-basedFactoidQA
CS295:STATISTICALNLP(WINTER2017) 10
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
Knowledge-basedQA(Siri)Buildasemanticrepresentationofthequery◦ Times,dates,locations,entities,numericquantities
Mapfromthissemanticstoquerystructureddataorresources◦ Geospatialdatabases◦ Ontologies(Wikipediainfoboxes,dbPedia,WordNet,Yago)◦ Restaurantreviewsourcesandreservationservices◦ Scientificdatabases
CS295:STATISTICALNLP(WINTER2017) 11
Hybridapproaches(Watson)Buildashallowsemanticrepresentationofthequery
GenerateanswercandidatesusingIRmethods◦ Augmentedwithontologiesandsemi-structureddata
Scoreeachcandidateusingricherknowledgesources◦ Geospatialdatabases◦ Temporalreasoning◦ Taxonomicalclassification
CS295:STATISTICALNLP(WINTER2017) 12
IBM’sWatson
WonJeopardy onFebruary16,2011!
13
WILLIAMWILKINSON’S“ANACCOUNTOFTHEPRINCIPALITIESOF
WALLACHIAANDMOLDOVIA”INSPIREDTHISAUTHOR’SMOSTFAMOUSNOVEL
BramStoker
CS295:STATISTICALNLP(WINTER2017)
MotivationforWatson
CS295:STATISTICALNLP(WINTER2017) 14
SingleSourceisnotSufficient
CS295:STATISTICALNLP(WINTER2017) 15
WatsonArchitecture
CS295:STATISTICALNLP(WINTER2017) 16
. . .
Models
Answer & Confidence
Question
Evidence
Sources
Models
Models
Models
Models
ModelsPrimarySearch
CandidateAnswer
Generation
Answer Sources Evidence
RetrievalEvidence Scoring
Learned Modelshelp combine and
weigh the Evidence
Hypothesis and Evidence Scoring
Merging &
Ranking
Synthesis
WatsonPerformance
CS295:STATISTICALNLP(WINTER2017) 17
Outline
QuestionAnswering
TextualEntailment
CS295:STATISTICALNLP(WINTER2017) 18
IR-BasedQASystem
OtherExtensions
IR-basedFactoidQA
CS295:STATISTICALNLP(WINTER2017) 19
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
IR-basedFactoidQAQUESTIONPROCESSING◦ Detectquestiontype,answertype,focus,relations◦ Formulatequeriestosendtoasearchengine
PASSAGERETRIEVAL◦ Retrieverankeddocuments◦ Breakintosuitablepassagesandrerank
ANSWERPROCESSING◦ Extractcandidateanswers◦ Rankcandidates
◦ usingevidencefromthetextandexternalsources
CS295:STATISTICALNLP(WINTER2017) 20
FactoidQ/A
CS295:STATISTICALNLP(WINTER2017) 21
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
QuestionProcessingAnswerTypeDetection◦ Decidethenamedentitytype(person,place)oftheanswer
QueryFormulation◦ ChoosequerykeywordsfortheIRsystem
QuestionTypeclassification◦ Isthisadefinitionquestion,amathquestion,alistquestion?
FocusDetection◦ Findthequestionwordsthatarereplacedbytheanswer
RelationExtraction◦ Findrelationsbetweenentitiesinthequestion
CS295:STATISTICALNLP(WINTER2017) 22
QuestionProcessing
AnswerType:USstate
Query:twostates,border,Florida,north
Focus:thetwostates
Relations:borders(Florida,?x,north)
CS295:STATISTICALNLP(WINTER2017) 23
They’rethetwostatesyoucouldbereenteringifyou’recrossingFlorida’snorthernborder
AnswerTypes:NamedEntities
WhofoundedVirginAirlines?PERSON
WhatCanadiancityhasthelargestpopulation?CITY
CS295:STATISTICALNLP(WINTER2017) 24
PartofAnswerTypeTaxonomy
CS295:STATISTICALNLP(WINTER2017) 25
LOCATION
NUMERIC
ENTITY HUMAN
ABBREVIATIONDESCRIPTION
country city state
datepercent
money
sizedistance
individual
title
group
food
currency
animal
definition
reason expression
abbreviation
Li,Roth.LearningQuestionClassifiers.COLING(2002)
AnswerTypes
CS295:STATISTICALNLP(WINTER2017) 26
MoreAnswerTypes
CS295:STATISTICALNLP(WINTER2017) 27
AnswertypesinWatson
2500answertypesin20,000Jeopardyquestionsample◦ Themostfrequent200answertypescover<50%ofdata
The40mostfrequentJeopardyanswertypes
he,country,city,man,film,state,she,author,group,here,company,president,capital,star,novel,character,woman,river,island,king,song,part,series,sport,singer,actor,play,team,show,actress,animal,presidential,composer,musical,nation,book,title,leader,game
CS295:STATISTICALNLP(WINTER2017) 28Ferrucci etal.BuildingWatson:AnOverviewoftheDeepQA Project.AIMagazine.2010
AnswerTypeDetection
CS295:STATISTICALNLP(WINTER2017) 29
Hand-writtenRules
Regularexpression-basedrulescangetsomecases:◦ Who{is|was|are|were}PERSON
Otherrulesusethequestionheadword:◦ (theheadwordofthefirstnounphraseafterthewh-word)◦ WhichcityinChinahasmostforeignfinancialcompanies?◦ WhatisthestateflowerofCalifornia?
MachineLearning
QuestionwordsandphrasesPart-of-speechtagsParsefeatures(headwords)NamedEntitiesSemanticallyrelatedwords
FactoidQ/A
CS295:STATISTICALNLP(WINTER2017) 30
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
KeywordSelectionAlgorithm1.Selectallnon-stopwordsinquotations2.SelectallNNPwordsinrecognizednamedentities3.Selectallcomplexnominals withtheiradjectivalmodifiers4.Selectallothercomplexnominals5.Selectallnounswiththeiradjectivalmodifiers6.Selectallothernouns7.Selectallverbs8.Selectalladverbs9.SelecttheQFWword(skippedinallprevioussteps)10.Selectallotherwords
CS295:STATISTICALNLP(WINTER2017) 31Moldovan,Harabagiu,Pasca,Mihalcea,Goodrum,GirjuandRus.TREC(1999)
Choosingkeywords
CS295:STATISTICALNLP(WINTER2017) 32
Whocoinedtheterm“cyberspace”inhisnovel“Neuromancer”?
1 1
4 4
7
cyberspace/1Neuromancer/1term/4novel/4coined/7
FactoidQ/A
CS295:STATISTICALNLP(WINTER2017) 33
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
PassageRetrieval
RetrievedocumentsusingIR◦ querytermsaskeywords
CS295:STATISTICALNLP(WINTER2017) 34
Step1
Step2Segmentthedocumentsintoshorterunits◦ somethinglikeparagraphs
Step3Passageranking◦ Useanswertypetohelprerank passages
FeaturesforPassageRanking
NumberofNamedEntitiesoftherighttypeinpassage
Numberofquerywordsinpassage
NumberofquestionN-gramsalsoinpassage
Proximityofquerykeywordstoeachotherinpassage
Longestsequenceofquestionwords
Rankofthedocumentcontainingpassage
CS295:STATISTICALNLP(WINTER2017) 35
Eitherinrule-basedclassifiersorwithsupervisedmachinelearning
FactoidQ/A
CS295:STATISTICALNLP(WINTER2017) 36
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
AnswerExtractionRunananswer-typenamed-entitytaggeronthepassages◦ Eachanswertyperequiresanamed-entitytaggerthatdetectsit◦ IfanswertypeisCITY,taggerhastotagCITY◦ CanbefullNER,simpleregularexpressions,orhybrid
Returnthestringwiththerighttype:◦ WhoistheprimeministerofIndia(PERSON)◦ ManmohanSingh,PrimeMinisterofIndia,hadtoldleftleadersthatthedealwouldnotberenegotiated.
◦ HowtallisMt.Everest?(LENGTH)◦ TheofficialheightofMountEverestis29035feet
CS295:STATISTICALNLP(WINTER2017) 37
RankingCandidateAnswersButwhatiftherearemultiplecandidateanswers!
Q:WhowasQueenVictoria’ssecondson?
AnswerType:Person
CS295:STATISTICALNLP(WINTER2017) 38
• Passage:TheMariebiscuitisnamedafterMarieAlexandrovna,thedaughterofCzarAlexanderIIofRussiaandwifeofAlfred,thesecondsonofQueenVictoriaandPrinceAlbert
RankingCandidateAnswersButwhatiftherearemultiplecandidateanswers!
Q:WhowasQueenVictoria’ssecondson?
AnswerType:Person
CS295:STATISTICALNLP(WINTER2017) 39
• Passage:TheMariebiscuitisnamedafterMarieAlexandrovna,thedaughterofCzarAlexanderIIofRussiaandwifeofAlfred,thesecondsonofQueenVictoriaandPrinceAlbert
FeaturesforMLAnswertypematch:Candidatecontainsaphrasewiththecorrectanswertype.
Patternmatch:Regularexpressionpatternmatchesthecandidate.
Questionkeywords:#ofquestionkeywordsinthecandidate.
Keyworddistance:Distanceinwordsbetweenthecandidateandquerykeywords
Noveltyfactor:Awordinthecandidateisnotinthequery.
Appositionfeatures:Thecandidateisanappositivetoquestionterms
Punctuationlocation:Thecandidateisimmediatelyfollowedbyacomma,period,quotationmarks,semicolon,orexclamationmark.
Sequencesofquestionterms:Thelengthofthelongestsequenceofquestiontermsthatoccursinthecandidateanswer.
CS295:STATISTICALNLP(WINTER2017) 40
ScoringCandidatesinWatsonEachcandidateanswergetsscoresfrom>50components◦ (fromunstructuredtext,semi-structuredtext,triplestores)◦ logicalform(parse)matchbetweenquestionandcandidate◦ passagesourcereliability◦ geospatiallocation◦ Californiais”southwestofMontana”
◦ temporalrelationships◦ taxonomicclassification
CS295:STATISTICALNLP(WINTER2017) 41
Outline
QuestionAnswering
TextualEntailment
CS295:STATISTICALNLP(WINTER2017) 42
IR-BasedQASystem
OtherExtensions
AskMSR
CS295:STATISTICALNLP(WINTER2017) 43
1 2
3
45
QuestionProcessing Search
AnswerExtractionAnswerScoring
Dumais, Banko, Brill, Lin, Ng,SIGIR(2002)
Step1:RewriteQueries
CS295:STATISTICALNLP(WINTER2017) 44
Intuition: Questions are often syntactically quite closeto sentences with the answer• Where istheLouvreMuseumlocated?
• TheLouvreMuseumislocated in Paris• Who createdthecharacterofScrooge?
• Charles DickenscreatedthecharacterofScrooge
FeedbackLoops:FALCON
CS295:STATISTICALNLP(WINTER2017) 45
AllenAIScienceChallenge
CS295:STATISTICALNLP(WINTER2017) 46
Whichobjectisthebestconductorofelectricity?(A) awaxcrayon(B)aplasticspoon(C)arubbereraser(D)anironnail
AllenAIScienceChallenge
CS295:STATISTICALNLP(WINTER2017) 47
Whichobjectisthebestconductorofelectricity?(A) awaxcrayon(B)aplasticspoon(C)arubbereraser(D)anironnail
AllenAIScienceChallenge
CS295:STATISTICALNLP(WINTER2017) 48
Fourthgradersareplanningaroller-skaterace.Whichsurfacewouldbethebestforthisrace?(A)gravel(B)sand(C)blacktop(D)grass
AllenAIScienceChallenge
CS295:STATISTICALNLP(WINTER2017) 49
Fourthgradersareplanningaroller-skaterace.Whichsurfacewouldbethebestforthisrace?(A)gravel(B)sand(C)blacktop (D)grass
§ Informationretrievalmethodsfail§ Wordco-occurrencemethodsstruggle
Graders are commonly used in the construction and maintenance of dirt roads and gravel roads
Also strong correlations between:grass « racegravel « surface
AllenAIScienceChallenge
CS295:STATISTICALNLP(WINTER2017) 50
Astudentputstwoidenticalplantsinthesametypeandamountofsoil.Shegivesthemthesameamountofwater.Sheputsoneoftheseplantsnearasunnywindowandtheotherinadarkroom.Thisexperimenttestshowtheplantsrespondto
(A)light(B)air(C)water(D)soil
AllenAIScienceChallenge
CS295:STATISTICALNLP(WINTER2017) 51
Astudentputstwoidenticalplantsinthesametypeandamountofsoil.Shegivesthemthesameamountofwater.Sheputsoneoftheseplantsnearasunnywindowandtheotherinadarkroom.Thisexperimenttestshowtheplantsrespondto
(A)light (B)air(C)water(D)soil
Knowledge needed(forexample):§ nearsunnywindow® receivelight§ inadarkroom® nolight§ testX’sresponsetoY
® compareX+YwithX+not(Y)
Outline
QuestionAnswering
TextualEntailment
CS295:STATISTICALNLP(WINTER2017) 52
IR-BasedQASystem
OtherExtensions
NaturalLanguage&Meaning
CS295:STATISTICALNLP(WINTER2017) 53
Meaning
Language
AmbiguityVariability
interpretatio
n expression
InferencevsEntailment
CS295:STATISTICALNLP(WINTER2017) 54
MeaningRepresentation
Natural Language
Inference
TextualEntailment
TextualEntailment
CS295:STATISTICALNLP(WINTER2017) 55
• Adirectionalrelationbetweentwotextfragments:Text(t) andHypothesis(h):
t entails h (t Þ h) if humans reading t will infer that h is most likely true
• Assuming“commonbackgroundknowledge”–whichisindeedexpectedfromapplications
Example
CS295:STATISTICALNLP(WINTER2017) 56https://cogcomp.cs.illinois.edu/page/resource_view/9
MoreSentencePairs
CS295:STATISTICALNLP(WINTER2017) 57
1. Some students came to school by car.Some students came to school.
2. No students came to school by car. Some students came to school.
3. John drove legally. John drove.
4. John drove predictably.John drove.
5. Legally, John could drive.John drove.
EntailmentwithKnowledge
CS295:STATISTICALNLP(WINTER2017) 58
Fortextualentailmenttoholdwerequire:◦ text ANDknowledgeÞ h,but◦ knowledgeshouldnotentailh alone
Systemsare notsupposedtovalidateh’struthregardlessoft(e.g.bysearchinghontheweb)
t entails h (t Þ h) if humans reading t will infer that h is most likely true
Example
CS295:STATISTICALNLP(WINTER2017) 59
TEXT: …WhilenooneaccusesMadonnaofdoinganythingillegalinadoptingthe4-year-oldgirl,reportedlynamedMercy,therearequestionsnonethelessabouthowMadonnaisabletonavigateMalawi's18-to-24monthvettingperiodinjustamatterofdaysorweeks…
HYPOTHESIS:
Madonnais50yearsold.
ThirdLabel:Contradictions
CS295:STATISTICALNLP(WINTER2017) 60
t contradicts h (t ^ h) if humans reading t will find the relations/events described by h to be highly unlikely given t.
Example
CS295:STATISTICALNLP(WINTER2017) 61
MoreExamples
CS295:STATISTICALNLP(WINTER2017) 62
Applications
CS295:STATISTICALNLP(WINTER2017) 63
QuestionAnswering
InformationExtraction
MachineTranslation
InformationRetrieval
Question Expected answer formWho bought Overture? >> X bought Overture
Overture’s acquisitionby Yahoo
Yahoo bought Overture
text hypothesized answer
entails
Heabhorredthemen’sunctuousways.
Hedislikedthemen’sflatteringways.
Similaritybasedonwhetherdocumententailsthequery.