AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: IBM Question Answering Update piQuAnt ARDA/AQUAINT...

AQUAINTAQUAINT IBMIBMPIQUANTPIQUANT

ARDAARDA CYCORPCYCORP

Subcontractor:

IBM Question Answering UpdatepiQuAnt

ARDA/AQUAINTDecember 2002 Workshop

This work was supported in part by the Advanced Research and Development Activity (ARDA)'s Advanced Question Answering for Intelligence (AQUAINT) Program under contract number MDA904-01-C-0988.

IBM Research

Subcontractor: Cycorp

IBM - PIQUANTIBM - PIQUANT

Overview

Progress with QPlans

Multi-Agent, Multi-Source Architecture & Answer Resolution

2002 Performance Evaluation

IBM Research



Single Strategy -> Plan-Based

More sophisticated Question Analysis using full parse and NE recognitionSearch strategy based on type of question. Any or all of following:

Regular – Predictive AnnotationRelative – Relative clauses, appositionsDefinition – Use external structured knowledge (WordNet, Tables from WWW, Databases, Cyc)

Corpus strategy (selected by user)Answering Agent strategy (selected by user)

IBM Research



A Single-Agent, Single-Strategy QA Architecture

QuestionQuestion

AnswerAnswer AnswerPresentation

AnswerPresentation

Answers

NLPUtilities

NLPUtilities

AnswerClassification


QuestionAnalysisQuestionAnalysis

SearchSearchHit Hit ListList

WordNetWordNet

AnswerSelectionAnswer

Selection

SE Query

Answer Type

IBM Research



A Multi-Agent QA Architecture

KSP-BasedAnswering Agent

Rule-BasedAnswering Agent

Answering Agents

StatisticalAnswering Agent

Definitional QAnswering Agent

QuestionQuestion

AnswerResolution

AnswerResolutionAnswerAnswer

QGoals

AnswerJustification

& Presentation

AnswerJustification

& Presentation

Answers

QFrame

NLPUtilities

NLPUtilities





WordNetWordNet

CycCyc


Selection

Web

KS Adaptation Layer

Web-BasedAnswering Agent

QPlanGenerator

QPlanGenerator

QPlanExecutorQPlan

Executor

AQUAINTAQUAINT IBMIBMPIQUANTPIQUANT

ARDAARDA CYCORPCYCORP

Subcontractor:

Merging and Resolving Answers in a Multi-Agent QA Architecture

Jennifer Chu-Carroll

November 8, 2002

IBM Research



A Multi-Agent QA Architecture

KSP-BasedAnswering Agent

Rule-BasedAnswering Agent

Answering Agents

StatisticalAnswering Agent

Definitional QAnswering Agent

QuestionQuestion

AnswerResolution

AnswerResolutionAnswerAnswer

QGoals

AnswerJustification

& Presentation

AnswerJustification

& Presentation

Answers

QFrame

NLPUtilities

NLPUtilities





WordNetWordNet

CycCyc


Selection

Web

KS Adaptation Layer (OntASK)

Web-BasedAnswering Agent

TREC 10TREC 11EB

TREC 10TREC 11EB

IBM Research



Currently Implemented Answering AgentsAgents based on unstructured information

Agent strategies:Knowledge-based answering agentStatistical answering agent

Knowledge sources:AQUAINT corpusTREC corpusEncyclopedia Britannica

Agents based on structured informationAgent strategies:

Knowledge source query via KSPSanity checking: Post-hoc filtering of candidate answers

Knowledge sourcesWordNetCycDatabases

IBM Research



Answer ResolutionCombine answers from multiple answering agents

QuestionAnalysis 1Question

Analysis 1Search 1Corpus 1Search 1Corpus 1

AnswerSelection 1

AnswerSelection 1


Analysis 2 Search 2Search 2 AnswerSelection 2

AnswerSelection 2

ConfidenceReranking

ConfidenceReranking

Answer Resolution

passages answers


Analysis 1Search 1Corpus 2Search 1Corpus 2

AnswerSelection 1

AnswerSelection 1

Agent 1

Agent 2

Agent 3

SanityChecking

SanityChecking

Final answer

IBM Research



Answer Resolution ComponentsAnswer Selection

Combines answers proposed by passages retrieved Using different keywords and/or search strategiesFrom different corpora using the same strategy

MotivationDifferent strategies/corpora may produce different relevant passagesSemantically-equivalent answers appearing in different contexts

Enables answer selection to better find close matches with questionReinforces one another

A corpus may bePrimary corpus: answers can be proposed and justifiedSupporting corpus: answers can only support those found in the primary corpus

IBM Research



Answer Selection Process

Identifies candidate answers and their semantic types

Evaluates candidate answers based onSemantic type match

Grammatical relationship match

Performs candidate answer normalizationE.g., Clinton = Bill Clinton = President Clinton

Currently focuses on named entity normalization

Combines evidence for each candidate answer and computes score

IBM Research



Answer Selection Example

TREC11 Q: “How many chromosomes does a human zygote have?”

Passages from rule-based strategy + AQUAINT corpus

Of the 46 human chromosomes, 44 are identical pairs.

There are 46 paired chromosomes in a human being’s cell nucleus.

… the order of the 21st of the 23 pairs of human chromosome, …

… narrowed their search of the gene to a small section of human

chromosome 7, …

… fused together to form the present-day human chromosome 7.

System returns “7” as its top answer

IBM Research



Answer Selection Example (Cont’d)

TREC11 Q: “How many chromosomes does a human zygote have?”

Passages from statistical strategy + AQUAINT corpus

… sequence the roughly 100,000 genes on the 46 human chromosomes.

Of the 46 human chromosomes, 44 are identical pairs.

Passages from rule-based strategy + TREC corpus

There are 46 chromosomes in a normal human cell.

… located on one of the 46 chromosomes in every human cell.

Passages from rule-based strategy + Encyclopedia Britannica

In each body cell of normal human beings, there are 46 chromosomes, …

Normally, humans have 46 chromosomes arranged in 23 pairs.

With additional passages, system now returns “46” as its top answer

IBM Research



Answer Resolution ComponentsConfidence Reranking

Invoked only if two or more strategically-independent answering agent are usedMotivation

Better confidence in the same answer given by two strategically-independent agents

ProcessAdjust confidence scores of previously-determined answers in consultation with another answer setScore receives large boost if identical answer given by other answer agentScore receives small boost if partially overlapping answer given

IBM Research



Answer Resolution ComponentsCyc Sanity Checker

A post-hoc process for Rejecting “insane” answers

How much does a grey wolf weigh? 300 tons

Boosting confidence for “sane” answers

Sanity checker invoked withPredicate, e.g. “weight”Focus, e.g. “grey wolf”Candidate value, e.g. “300 tons”

Sanity checker returns“Sane”: + or – 10% of value in Cyc“Insane”: outside of the reasonable range“Don’t know”

Confidence score highly boosted when answer is “sane”

IBM Research



Cyc Sanity Checking Example

Trec11 Q: “What is the population of Maryland?”

Without sanity checkingPIQUANT’s top answer: “50,000”

Justification: “Maryland’s population is 50,000 and growing rapidly.”

Passage discusses an exotic species “nutria”, not humans

With sanity checkingCyc knows the population of Maryland is 5,296,486

It rejects the top “insane” answers

PIQUANT’s new top answer: “5.1 million” with very high confidence

IBM Research



Performance EvaluationConducted experiments to evaluate the multi-source and multi-strategy aspects of PIQUANT System configurations

TREC2001 system: pre-AQUAINTSingle Source & Single Strategy:

Strategy: Rule-based or statistical answering agentSource: AQUAINT corpus

Multiple Sources & Single Strategy:Strategy: Rule-based answering agentSources

Primary: AQUAINT corpus Supporting: TREC corpus, EB

Multiple Sources & Multiple Strategies:Strategies: Rule-based and statistical answering agentsSources

Primary: AQUAINT corpusSupporting: TREC corpus, EB

IBM Research



Evaluation Results

Overall impact of agents based on unstructured information41.3% relative improvement in number of questions correctly answered51.8% relative improvement in average precision

Impact of agents based on structured knowledge sourcesKSP invoked 5 times, returned 5 correct answersCyc sanity checker invoked 3 times, returned 1 definitive answer

TREC2001system

SinglesourceRule-based strategy

SinglesourceStatistical strategy

Multiple sourcesRule-basedstrategy

MultiplesourcesMultiplestrategies

% correct

28.3% 32.5% 32.7% 38.2% 40.0%

Avg prec 0.446 0.531 0.534 0.622 0.677

AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: IBM Question Answering Update piQuAnt ARDA/AQUAINT...

Documents

Transcript of AQUAINT IBM PIQUANT ARDACYCORP Subcontractor: IBM Question Answering Update piQuAnt ARDA/AQUAINT...