Inducing Relations

Inducing Relations1Document 1... Boston was founded on November 17, 1630, by Puritan colonists from England ...Document 2Document 3... New York City was settled by Europeans from The Netherlands in 1624 ...... San Francisco was founded in 1776 by the Spanish conquerors ...Goal: Discover types of information salient to a domain and extract short phrases representing themWere interested in automatically inducing the structure of documents.By structure, we mean how content is selected and ordered across related documents from a single domain.We learn this structure from raw unannotated documents, without any information about section markers or headings.For example, consider a set of encyclopedia articles about cities.Wed like to automatically induce a set of topic assignments for different portions of the documents.Portions assigned the same topic (denoted by the same color here) should talk about semantically related content.

1Application: Creating MetadataMachine-readable access mechanism for searching, browsing, and retrieving text2MedicalRecordsLocationRotator cuffSeverityMildTear Length2mmDisasterReportsInjuriesNoneLocationMelbourne, FLTimeTuesday morningApplication: Generating InfoboxesExploring the important attributes of new domains automatically3

CambridgeSeattleRegularities for Learning RelationsLocal lexical and orthographic similarity in expression of relation instances4Recurring specific syntactic patterns in relation occurrence injured six people injured 16 relief workers four were hurt{number}injur*Evoking wordRelation phraseSimilar document-level positioning of relations

injuredkilledsixThreeNote this is the example for the document-level stuff for the slide above5A strong earthquake with a magnitude of 5.6 rocked the easternmost province of Irian Jaya on Friday.

An earthquake of magnitude 6 is considered severe, capable of widespread damage near the epicenter.Beginning of DocumentEnd of DocumentHighlights of the ApproachNovel source of supervision: declarative human knowledge about constraintsRich combination of information sources that combines multiple layers of linguistic analysisMathematical formalism that guides unsupervised learning with human knowledge

Indicators: is_verb010earthquake100hit010......Arguments: has_capital011is_number000height112......

Input Representation Each potential indicator word and argument phrase encoded with features8injuredVBNsixpeople+ discourse contextVPSNPsyntactic contextOutput Representation Relation instances as indicator word and argument phrase pairings.Our Two-Pronged ApproachModel Structure: A generative model of hidden indicator and argument structureModels local lexical and syntactic similarityBiases toward consistent document-level structure

Soft declarative constraints: Enforced during inference via posterior regularizationRestricts global syntactic patterns Enforces relation instance uniqueness9

Generating Indicators and ArgumentsFor a single relation type, indicators and arguments drawn from relation-specific feature distributions1010

: parameters of indicator feature distributions: parameters of argument feature distributions

Backoff DistributionsRemaining constituents generated from backoff feature distributions1111: parameters of indicator feature distributions: parameters of argument feature distributions

Multiple Relations (maybe delete if no fit)12Each relation has its own Constituent features drawn from pointwise product over all : either indicator or backoff for each : either argument or backoff for each

Selecting Relation LocationsRelation instance locations within document drawn from shared distributionIndicator and argument within sentence selected uniformly at random

13Document 1

Sentence 1.Sentence 2.Sentence 3.Sentence 4.Document 2

Sentence 1.Sentence 2.Document 3

Sentence 1.Sentence 2.Sentence 3.Document 4

Sentence 1.Sentence 2.Sentence 3.

Summary of Generative ProcessFor each relation :Draw indicator, argument, and backoff distributions:Draw location distribution:

14

(continuation of previous slide)For each document :For each relation :Select a sentence (or null): Draw argument and indicator positions uniformly at random within sentence For each potential indicator word :Draw indicator features: is if this word selected as indicator, otherwiseFor each potential argument phrase :Draw argument features:

15

Model PropertiesLexical similarityVia featuresRecurring syntactic patternsVia features and constraints during learningRegularities in document-level structureVia document location distributionIssue: how do we break symmetry between relations?Via constraints during learning

16Variational Inference with Declarative ConstraintsDesired posterior:

Optimize variational objective with mean field factorization:

Model parametersHidden structure (relations)Observed data (words, trees)

17Syntactic Constraints18Counts number of relationsin that match canonicalsyntactic pattern

Biases toward relations that are syntactically plausibleIndicator is verb and argument is object of indicatorIndicator is noun and argument is modifierIndicator and argument are subject/object of same verb

Threshold of relations that mustmatch syntactic pattern (80%)Separation Constraints (Argument)19Counts number of relationswhose arguments includeword : no more than one relationshould share the same argument wordEncourages relations to be diverseArguments cannot be shared...

Separation Constraints (Indicator)20Counts number of relationswhose indicators includeword : allow some relationsto share indicator wordsEncourages relations to be diverseIndicators can be shared to an extent

Experimental SetupExperiments on two news domains

Example Document

21CorpusNumber of DocumentsSentences/ DocumentWords/DocumentRelation TypesFinance10012.1262.915Earthquake2009.3210.39A strong earthquake rocked the Philippines island of Mindoro early Tuesday, killing at least two people and causing some damage, authorities said. The 3:15 am quake had a preliminary magnitude of 6.7 and was centered near Baco on northern Mindoro Island, about 75 miles south of Manila, according to the Philippine Institute of Vulcanology and Seismology. The U.S. Geological Survey in Menlo Park, Calif., put the quake's preliminary magnitude at 7.1. Gov. Rodolfo Valencia of the island's Oriental Mindoro province said two people reportedly were killed and that several buildings and bridges were damaged by the quake. Several homes near the shore reportedly were washed away by large waves, Valencia told Manila radio station DZBB. Telephone service was cut, he said. The quake swayed tall buildings in Manila. Institute spokesman Aris Jimenez said the quake occurred on the Lubang fault, one of the area's most active. A magnitude 6 quake can cause severe damage if centered under a populated area, while amgnitude 7 quake indicates a major quake capable of widespread, heavy damage.

Say something about this at beginning.21Extracted RelationsLocation

Magnitude22A strong earthquake rocked the Philippines island of Mindoro early Tuesday, killing at least two people and causing some damage, authorities said. The 3:15 am quake had a preliminary magnitude of 6.7 and was centered near Baco on northern Mindoro Island...The 3:15 am quake had a preliminary magnitude of 6.7 and was centered near Baco on northern Mindoro Island...TimeGeneric versus Domain-specific KnowledgeGeneric Feature RepresentationIndicator: word, POS, word stemArgument: word, syntax label, headword of parent, dependency label to parent

Domain-specific knowledge (relation independent)Finance: prefer arguments with numbersEarthquake: prefer relations in first two sentences of each document23Main Results (Sentence F-score)FinanceEarthquake USP: Unsupervised semantic parsing(Poon and Domingos 2009) CLUTO: CLUTO sentence clustering Mallows: Mallows content model sentence clustering (Chen et al 2009)Main Results (Token F-score)25FinanceEarthquake USP: Unsupervised semantic parsing(Poon and Domingos 2009)Constraint Ablation AnalysisWhat happens as we modify declarative constraints?No-sep:No separation constraintsNo-syn:No syntactic constraintsHard-syn:Always enforce syntactic constraints26FinanceEarthquakeWhat if we had Annotated Data?27