Question Answering
-
Upload
miranda-harvey -
Category
Documents
-
view
46 -
download
0
description
Transcript of Question Answering
![Page 1: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/1.jpg)
18-12-07 | 1/21
› Off-line answer extraction / Jori Mur
Question Answering
![Page 2: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/2.jpg)
18-12-07 | 2/21
Overview› Using patterns to extract answers off-line› High precision, low recall
• Simple patterns not robust enough• Answer not in one sentence• No binary relation in question• Hard to come up with new patterns
![Page 3: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/3.jpg)
18-12-07 | 3/21
Joost
![Page 4: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/4.jpg)
18-12-07 | 4/21
Off-line answer extraction using patterns
› /[Noun] [Name]/... premier Balkenende ...
› /[Name], [Noun]/... Balkenende, premier ...
› /[Name] [Verb] as [Noun]/... Balkende heeft als premier ...
› /[Name] [be] [Det] [Noun]/... Balkenende is de premier ...
› ...
![Page 5: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/5.jpg)
18-12-07 | 5/21
High precision, low recall
› Tjong Kim Sang, Bouma en de Rijke. Developing Offline Strategies for Answering Medical Questions, 2005
› Bernardi, Jijkoun, Mishne en de Rijke.Selectively Using Linguistic Resources Throughout the Question Answering Pipeline,2003
![Page 6: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/6.jpg)
18-12-07 | 6/21
Investigating experiment› Joost› 99 function questions from CLEF2003 and
CLEF2004• Who was the author of Die Gotterdämmerung?• Of which company is Christian Blanc president?
› 15 patterns• /[ADJ][NOUN][NAME]/
Franse president Sarkozy• /[NAME], [NOUN] van [COUNTRY]/
Sarkozy, president van Frankrijk
![Page 7: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/7.jpg)
18-12-07 | 7/21
Results› Extraction
› QA
Judgement # extracted factsCorrect 58Partially correct 15Incorrect 27Total 100
Judgement # questionsCorrect 35No answer found 46Incorrect 18Total 99
![Page 8: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/8.jpg)
18-12-07 | 8/21
Error analysis: 46 No answer found
› In which company is Christian Blanc president?
› Christian Blanc, president of the rather destitute French aviation company Air France
Reason # QuestionsAnswers could not be found 14Patterns don't match 20Reference resolution needed 6Other 6Total 46
› Who is the daughter of Deng Xiaopeng?› Will Deng Xiaoping make it until the
Chinese New Year? (...) when Deng's youngest daughter, Deng Rong, recently said (...)
› Who is the head of state of Australia?› The prime-minister of Australia explained his
plan to transfer the role of the head of state from the British queen Elizabeth to (...)
![Page 9: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/9.jpg)
18-12-07 | 9/21
Error analysis: 18 Incorrect
› Who is the German minister of Economy?› Who was the president of the US during the
second World War?
Reason # QuestionsModifiers ignored 12Other 6Total 18
![Page 10: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/10.jpg)
18-12-07 | 10/21
4 Challenges
› Simple patterns are not robust enough
› Not easy to think of new patterns
› Answer not in one sentence
› Not always binary relations
![Page 11: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/11.jpg)
18-12-07 | 11/21
1. Simple patterns are not robust enough
... George Bush, yesterday he became 61, said today as 43rd president of the US ...
... George Bush said as president of the US ...
PredmSubj Obj ObjMod
Subj Predm Obj ObjMod
› <said, subj, George Bush>
› <said, predm, as>
› <as, obj, president>
› <president, mod, of>
› <of, obj, US>
![Page 12: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/12.jpg)
18-12-07 | 12/21
2. Answer not in one sentence
Balkenende was happy with the result, allthough he did not win the elections.
![Page 13: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/13.jpg)
18-12-07 | 13/21
2. Evaluation and Results CoreferenceMUC score : |S|-|p(S)|
|S| -1
Recall: 45.6%
Precision: 67.9%
ExtractionWithout coreference With coreference127.711 facts 146.210 facts
QAWithout coreference With coreference153/233 (65.7%) 159/233 (68.2%)Question 20050107:
Wie was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde? ‘Who was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope?’.
The answer in the AD of September 19th, 1995:Bowersox was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde. ‘Bowersox was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope’.
![Page 14: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/14.jpg)
18-12-07 | 14/21
3. Not always binary relations› When was Jimi Hendrix born?
Jimi Hendrix – 1942
› Who is the president of Amerika?Bush – president – Amerika
› Who was the president of Amerika in 1944?Roosevelt – president – Amerika – 1944
› Who succeeded president Roosevelt as president of Amerika in the second world war?Truman – president – Amerika – WO II Truman – successor – Roosevelt
![Page 15: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/15.jpg)
18-12-07 | 15/21
3. Dependency relation overlap Q: Who succeeded president Roosevelt as president of
Amerika in the second world war?A1: George Bush, president of Amerika said that ...A2: At the end of the second world war Harry S. Truman succeeded president Roosevelt as president of Amerika.
A1: <president, mod, of> <of,obj,Amerika>
A2: <succeed, subj, X> <of, obj, Amerika>
<succeed, obj, president Roosevelt> <world war, det, the>
<succeed, predm, as> <world war, mod, second>
<as, obj, president>
<president, mod, of>
![Page 16: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/16.jpg)
18-12-07 | 16/21
4. Hard to transfer to new question type› Learn patterns in bootstrapping way
1. Start with 10 seed-pairs2. Find dependency path between two terms3. Replace terms by variables to create pattern4. Use patterns to find new facts5. Use new facts to find new patterns
Find balance precision/recall3a. Filter patterns
![Page 17: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/17.jpg)
18-12-07 | 17/21
4. Calculate precision patterns› Replace only the answer term with a variable.
› P = Ca/Co› Co : the total number of times the pattern
matched a phrase in the corpus› Ca : the total number of times the pattern
matched a phrase in the corpus containing the correct answer term
![Page 18: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/18.jpg)
18-12-07 | 18/21
4. Experiment and Results learning› Stop at precision of table < 0.5 or
more than 100 patterns › 24 capital question from Clef, 3 turned out
NILFreq > 1 # patterns # facts P of facts QA
24 179127 <0 5 of 21
Freq > 1, P>=0.511 2403 0.76 16 of 21
143 3572 0.55 19 of 21
Freq > 1, P>=0.759 2375 0.75 16 of 21
93 2792 0.72 17 of 21125 3031 0.61 17 of 21
After 1st round
After 1st roundAfter 2nd round
After 1st roundAfter 2nd roundAfter 3rd round
![Page 19: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/19.jpg)
18-12-07 | 19/21
4. Nice examples of found patterns› ... is na H de grootste stad van L...
Petersburg is na Moskou de grootste stad van Rusland
› ... de L-adj tenniskampioenschappen in H...de open Franse tenniskampioenschappen in Parijs
› ... de L-adj president opent in H...De Tjechische president Havel opent op 14 juni in Praag
› ... de L-adj ministers komen bijeen in H...Volgende maand komen de Europese minister van industrie opnieuw bijeen in Brussel
![Page 20: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/20.jpg)
18-12-07 | 20/21
Summary› Qatar – off-line answer extraction module in
Joost
High precision, low recall
• Dependency patterns• Coreference resolution• Score for dependency overlap• Learning patterns
![Page 21: Question Answering](https://reader035.fdocuments.net/reader035/viewer/2022062422/56812bfc550346895d906f6b/html5/thumbnails/21.jpg)
18-12-07 | 21/21
Questions?